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[57] ABSTRACT 

The present invention is an integrated system for authoring, 
operating, maintaining, and administering a computer-based 
training program. The system includes integrated hardware 
and software that allows an author to create computer-based 
training modules that include audio, video, and textual 
content without requiring any programming. The invention 
uses a virtual training center metaphor that serves as the 
basis of the interface between the users and the system. The 
virtual training center provides a nonintimidating visual 
environment that is open and relaxing, thereby preparing the 
student to learn rapidly. To create a non-intimidating envi- 
ronment and to fix more of the student's attention on the 
subject matter to be learned, rather than on the learning tools 
themselves, the interface maintains as much as possible the 
illusion of being in a real training center. The system 
performs administrative functions, such as ensuring that 
students take courses in proper sequence, tracking training 
activity and course completion, and notifying the course 
administrator when a course is completed. 
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AUTHORING SYSTEM AND METHOD FOR 
COMPUTER-BASED TRAINING 

This application claims priority from U.S. Provisional 
patent application Ser. No. 60/052,792, filed Jul. 7, 1997. 

FIELD OF THE INVENTION 

Hie invention is related to the field of computer-based 
training and, in particular, to a system for authoring, 
administering, and deploying an interactive computer train- 
ing curriculum. 

BACKGROUND OF THE INVENTION 

Computer-based training has been used successfully by 
industry for many years because it provides many advan- 
tages over conventional instructor-led training classes. 
Unlike a classroom in which tbe students and instructor must 
meet physically in the same location at the same time, 
computer-based training can be made available at any time 
and place that is convenient for individual students. 
Computer-based training developed at one site can be used 
around the world. Unlike classroom instruction, in which the 
instructor must target training speed to the majority of 
students, bypassing slow learners and boring fast learners, 
computer based training is one-to-one, so that each indi- 
vidual can proceed at his or her own pace. 

Students retain more of what they learn in computer- 
based training because it is possible to reinforce tbe learning 
by having each individual student perform some task on the 
computer repeatedly to apply the newly acquired knowl- 
edge. Computer-based training also allows students to 
review their training skills as needed and when a refresher 
course is necessary, without tbe usual scheduling conflicts 
and the expense of an instructor. Current computer-based 
training systems suffer from several drawbacks. Computer- 
based training courses, particularly multimedia computer- 
based training courses, are very expensive to develop. One 
commonly used estimate for the cost of computer-based 
training is that each minute of training costs $1,000 to 
produce. This is based on approximately 2000 hours of work 
required to create a single hour of computer-based training 
at $30 per hour. 

Producing a computer-based module requires expertise in 
the course subject matter as well as expertise in technical 
areas such as recording audio and video information, inter- 
facing the recorded information with a computer, editing the 
audio and video information, page design, and educational 
theory. Merely figuring out how to digitize and input audio 
or visual data of a particular format into an editing program 
that requires a different format, can be a formidable task. 
Inferior computer-based training modules that are ineffec- 
tive result from computer-based training modules being 
produced by individuals that lack any of the many skills 
required to produce a good training module. 

Another problem with computer based training is the 
discomfort that many students feel with computers. Students 
are required to learn not only the subject matter, but also 
how to operate the computer and several applications with 
different look and feel, which can cause stress that reduces 
the effectiveness of learning. 

SUMMARY OF THE INVENTION 

An object of the invention is, therefore, to provide a 
training system that conveys a maximum knowledge to the 
student for a minimum of resources. 
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Another object of the invention is to provide a non- 
intimidating learning environment that minimizes the stu- 
dent's attention on the training tools, that is, the computer 
hardware and operating software, to maximize the student's 
5 attention on the subject matter being taught, to encourage the 
students and to ensure they can perform the trained tasks 
with ease. 

Yet another object of the invention is to remove technical 
barriers to the authoring of computer-based training so that 
to authors are not required to become computer engineers or 
programmers. 

Still another object of this invention is to provide training 
system administration that ensures students follow a pre- 
scribed curriculum and that faithfully records training activ- 
15 ity. 

Yet another object of the invention is to provide an 
integrated training system that provides standardized, 
compatible, extensible tools for authoring, learning, and 

M administering training. 

The present invention is an integrated system that 
includes functions for authoring, operating, and maintaining 
a computer-based training program. In one aspect of tbe 
invention, the system includes integrated hardware and 

25 software that allows an author to create computer-based 
training modules that include audio, video, and textual 
content without requiring any programming. The author is 
not required to be aware of tbe underlying operating system 
of the computer or of the details of interfacing between the 

30 authoring computer and the recording tools. The system is 
extensible in that the author, if so desires, can access the 
underlying operating system and programming tools to 
create training content that is beyond the capability of the 
original system. The system also allows tbe integration of 

35 other technologies, such as audio, video and imaging for- 
mats or other media, without requiring the author to perform 
hardware or software interfacing. 

Another aspect of tbe invention is the use of a training 
center metaphor that serves as the basis of the interface 

40 between the users and the system. Although the subject 
matter is presented on a computer terminal screen, images 
on the screen create the impression of a physical training 
center. Cues that indicate that a computer is being used are 
minimized or eliminated. The virtual training center pro- 

45 vides a nonintimidating visual environment that is open and 
relaxing, thereby preparing the student to learn rapidly. 

To create a non-intimidating environment and to fix more 
of the student's attention on the subject matter to be learned, 
rather than on the learning tools themselves, the interface 

50 maintains as much as possible the illusion of being in a real 
training center. To this end, manifestations of the operating 
system, such as computer icons and pop-up menus, are 
minimized or eliminated. By minimizing computer symbols, 
a student's discomfort with the computer can be reduced, 

55 and the student can concentrate on learning the material, not 
on learning how to operate and interact with the computer. 
All learning modules created with the invention have a 
similar look and feel so that the user can become comfort- 
able with a single interface that is used for training, 

60 authoring, and administering varied training courses and 
programs. 

To conform to the metaphor of an open, relaxing learning 
environment, the virtual training center may display, for 
example, an open room with training terminals partially 
65 walled off from each other to the extent that would be 
required to reduce distractions for real students. A large 
portion of the outside walls of the virtual training center may 
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be windowed, showing a quiet, relaxing scene, such as a 
serene mountain lake or a desert. The lack of computer 
symbols, such as icons and pull down menus serves to 
maintain the illusion of a real, physical training center: Real 
physical training centers do not have pull down menus. 

The system also performs administrative functions, such 
as ensuring that students take courses in proper sequence, 
tracking training activity and course completion, and noti- 
fying the course administrator when a course is completed. 

The subject matter of the present invention is particularly 
pointed out and distinctly claimed in the concluding portion 
of this specification. However, both the organization and 
method of operation, together with further advantages and 
objects thereof, may best be understood by reference to the 
following description taken in connection with accompany- 
ing drawings wherein like reference characters refer to like 
elements. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a screen display showing an image of a student 
at the Registration Board at the entrance of the 'virtual 
Training Center; 

FIG. 2 is a screen display showing an image of a student 
having entered the Virtual Training Center of FIG. 1; 

FIG. 3 is a screen display showing an image a workstation 
in the Virtual Training Center of FIG. 1 from the viewpoint 
of a student approaching the workstation; 

FIG. 4 is a screen display showing an image of two Virtual 
Interactive Players from the viewpoint of a student entering 
the workstation in the Virtual Training Center of FIG. 3; 

FIG. 5 is a screen display showing an image from the 
viewpoint of a student of leaving the Virtual Training Center 
of FIG. 1; 

FIG. 6 is a screen display showing an image from the 
viewpoint of a student having left the Virtual Training 
Center of FIG. 1; 

FIG. 7 shows a typical workstation used with the present 
invention; 

FIG. 8 is a screen display showing the Registration Board 
of FIG. 1; 

FIG. 9 is a screen display showing on the Virtual Inter- 
active Player of FIG. 4 a typical author's Virtual Interactive 
Player Directory 

FIG. 10 is a screen display showing on the Virtual 
Interactive Player of FIG. 4 a Request Form for communi- 
cating between system users; 

FIG. 11 is a screen display showing on the Virtual 
Interactive Player of FIG. 4 a typical review question 
presentation; 

FIG. 12 is a screen display on the Virtual Interactive 
Player of FIG. 4 showing a typical fishbone diagram; 

FIG. 13 is a screen display a showing on the Virtual 
Interactive Player of FIG. 4 a table of contents for admin- 
istrators; 

FIG. 14 is a screen display showing on the Virtual 
Interactive Player a typical student's Virtual Interactive 
Player Directory; and 

FIG. 15 shows the interrelationship of the hardware 
components of the present invention. 

DETAILED DESCRIPTION 

There are three user levels for logging onto the system: 
author, administrator, and student. The invention is most 
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clearly explained by describing the aspects common for all 
users and then the separate but integrated functions of the 
three different types of users. 

5 Common Aspects 

The preferred embodiment uses a metaphor of a training 
center to create a consistent, non-intimidating, relaxed learn- 
ing environment. FIGS. 1-6 show an example of how a 
Virtual Training Center (VTC) would appear to a user. The 

10 non-intimidating environment of the virtual training center 
metaphor is maintained for all three levels of users. 

The invention is preferably implemented using an indus- 
trial computer workstation with a large screen, high resolu- 
tion monitor. (FIG. 7) The display presents the training 

15 material on a portion of the computer terminal screen and 
shows the background of the virtual training center on the 
remainder of the screen. (FIGS. 8-14) For example, in a 
monitor that supports a resolution of 1024x768 pixels, the 
learning material is presented in a 640x480 pixel area and 

20 the remainder of the monitor displays a background showing 
a virtual training center. The constant display of the virtual 
training center in the background keeps the user psycho- 
logically connected to the virtual training center and helps 
prevent disorientation. 

Controls on the screen are designed to resemble mechani- 
cal devices. For example, the student plays a video clip of an 
operation by using a control that resembles a control on a 
familiar video cassette player. Sound effects, such as motors 

3Q turning and other mechanical sounds, are used to simulate 
the mechanical device to enhance the perception of operat- 
ing a real mechanical device. The function of each mechani- 
cal control, such as knobs and pushbuttons, is displayed on 
the device's display screen as a mouse cursor approaches a 

35 control. 

As a user begins a session, the monitor displays an image 
of a Registration Board (FIGS. 1 and 8) at the entrance of the 
Virtual Training Center. Th8^sys*sna. : identifies, the access 
level of a user as a student, aujhdi^or Qdminisirator as they 

40 log in using their 'assigned identification and password. The 
Registration Board can also provide a choice of languages 
for the training. The user is instructed, verbally and in text, 
in the log-in procedure in several languages. Selecting a 
particular language, can cue the Application to present the 

45 course in that particular language, provided it has been 
authored as such. If the user enters a valid identification and 
password, the display shows the perspective of an individual 
moving towards an entry door. The user then presses a 
button to enter the Virtual Training Center. By pressing the 

50 button for entry, the user is performing an affirmative act to 
psychologically prepare himself for authoring, training or 
administration. 

Upon entering the training area of the cylindrical Virtual 
Training Center, the Application shows a student's perspec- 

55 tive moving towards an Atrium at its center. The Virtual 
Training Center is shown in a 360° panoramic view, having 
exterior windows that look out upon a serene vista, such as 
a mountain or a desert scene. Even during actual training, a 
portion of the screen maintains the image of the Virtual 

60 Training Center, thereby maintaining the student in a slate of 
mind conducive to training by having him feel he is in a 
quiet, peaceful setting, as opposed to a noisy workplace. The 
user is then guided through the Atrium and proceeds to one 
of several training workstations, that is, a "Multimedia 

65 Instructor" containing a Virtual Interactive Player ("VIP'). 
At the VIP, the user is presented with the VIP Directory 
that shows the options accessible to the user. The options 
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will depend on the type of user and on the individual standards, such as the Intel lndeo, can also be used although 

student's curriculum. The VIP Directory is essentially a they typically do not support near real time compression, 

menu appearing in the form of a 5-part dynamic directory When a digitized video file is edited to produce video 

(FIG. 9). It indicates the options that are available to the user portions for a training module, the original file is saved so 

or that will be available to the user in the future, such as 5 that it can be called and edited again if necessary, 

courses that are part of the student's curriculum but are The author has the ability to place tests at different points 

unavailable until prerequisites are completed. The options 0 f mc course. For example, the author may place tests after 

are color coded: green options execute when they are cvcr y screen to test that the student has learned the infor- 

selected and white options open another set of sub-options. mat f on presented on each screen. The tests are typically 

The directory sub-options can go up to four deep. 10 ma( fc U p of questions that invite a student response that 

The selections run different applications. Each application requires a minimum amount of typing, such as true or false 

has a table of contents. Maintaining the metaphor of a questions or questions that require a number for the answer, 

physical environment, the table of contents of each appli- Such questions can be easily interpreted by the computer, 

cation appears as it would in a real book. The application The student will have a time limit, typically 20 seconds, for 

also includes an alphabetical list of topics. Both the Table of 15 answering each question on a test. After a question is posed, 

Contents and the List of Topics arc generated and updated a both an analog and a digital timer arc displayed, showing the 

automatically as the author assembles or edits a course. time remaining for answering the question and changing 

A common item to all users is a Request Form (FIG. 10), from green to yellow to red as the time expires. If the student 

that is, an internal messaging system by which the students, does not pass the test within the allowed time, the screen 

administrators, and authors can communicate about training 20 containing the material that is being reviewed is redisplayed 

issues. The messaging system is not interconnected with any and the student has another opportunity to learn the infor- 

other network, so that only training related messages are mation. The student is sent back after failing a test question 

passed through the system, thereby removing non-training as often as necessary until the student learns the information 

related distractions during training sessions. and can answer all questions correctly. 

To be consistent with the virtual environment, when a user 25 After each section of the training material, the author can 

is ready to end the current session, he simply "pushes" to place an overview of what was presented in that section. 

POWER button to turn the VIP off. The Application then (FIG. 11) There can also be tests after a section of material 

shows the visual perspective of the user as he backs off the is presented. If the student answers any of the questions 

workstation, turns towards the exit doors of the VTC which incorrectly on the section test, the screen re-displays the 

open automatically letting external sounds of nature such as 30 page at which the information required to answer the ques- 

bird songs to be heard. (FIGS. 5 and 6) tion is introduced. This provides the student another oppor- 
tunity to learn the material, before returning to the section 

Authors test 

To help an author design and organize a course before 35 Similarly, there can be an overview at the end of the entire 

beginning to write and videotape substantive material, the course and then a test on the material presented in the entire 

System makes available to the author an Instructional course. Incorrect answers on the overall test will also cause 

Design pre-authoring tool, such as Designer's Edge from the screen in which the unlearned information was presented 

Allen Communications, Salt Lake City, Utah. to be displayed and then the student will return to the overall 

After designing and organizing a course, the author begins 40 test. Students have the opportunity to repeat the training 

to construct it by assembling the various media. To create a material as often as necessary to learn the subject matter, 

complex instruction for a particular operation, such as the Because the student's responses are not recorded by the 

calibration of a measuring instrument, an author may choose System, and incorrect answers are not observed by other 

to videotape the procedure instead of describing it in text and students or by an instructor, the student will be more relaxed 

illustrations. The video is then digitized and stored into the 45 anc! more able 10 assimilate information, 

computer. The author can divide the video into portion, Thus, the student is preferably questioned about the 

which can be as short as a single frame. Controls on the subject matter three times: immediately after it is learned, at 

display let the author move through the video frames for- the end of the section, and at the end of the training module, 

ward or backward to determine exactly which frames should The student typically must answer all of the questions 

be included in any individual step of the training. 50 correctly to be passed on to the next level. The number of 

Portions of the video are assigned step numbers and the questions at each level of review typically increases. Stu- 
author can associate audio information and text with each dents can go back in the course to review sections already 
step. Information is preferably passed to the student visually completed but cannot go forward out sequence. Once the 
or aurally, as opposed to textually, to improve retention. Text entire course is completed, the student can access the entire 
can be associated with different areas of a displayed image 55 course in any order much like a textbook becomes a refer- 
so that when the student moves the cursor over that portion ence after the course is completed. Of course, an author can 
of the image, the text is displayed. For example, when a arrange the review questions as he feels appropriate, 
machine is displayed, the display can show the names of One embodiment of the invention includes capabilities to 
parts of the machine as the cursor is moved over those parts. allow an author to easily create interactive "fishbone" 
By not displaying information until the cursor is over the 60 (Ishikawa) diagrams (FIG. 12). These type of diagrams are 
region of interest, the display is less cluttered and conveys very effective in troubleshooting by visually depicting the 
information more clearly to the students. relationships of cause and effect. All objects comprising the 

Various known video compression formats can be used. interactive diagram are pre -designed so that the author only 

For example, the MPEG standard using 3x real time com- needs to add symptoms and remedies to construct an effec- 

pression provides the advantage of near real time data 65 tive troubleshooting diagram. The student is guided visually 

compression with the flexibility of being able to edit the and systematically towards a solution. At the conclusion of 

digitized material. Other video capture and compression a troubleshooting session, the date, time, user's name, 
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symptom and actions taken are recorded in the Trouble- 
shooting Log database. The database can then be used to 
derive statistical information pertaining to a particular sys- 
tem such as frequency and type of failures by system or 
component. 

An author can access screens of the program out of 
sequence, whereas a student must follow the prescribed 
order. An author is also allowed to access the underlying 
operating system for finding files or other administrative 
functions. One implementation of the invention uses the 
Window NT® operating system from Microsoft 
Corporation, Redmond, Wash. Other operating systems may 
also be used. 

In the Author mode, the VIP has an additional button that 
the student never sees. When engaged, this button elevates 
the Editing Menu to full view. The menu includes several 
editing functions that the author can use to add, delete and 
modify the course content. All this is accomplished without 
programming. When disengaged, the EDIT button retracts 
the Editing Menu below the table top that supports the VIP 
and the Table of Contents as well as the Index of Topics are 
updated automatically. 

One implementation of the inventive system was written 
primarily in ToolBook II from Asymetrix Corporation, 
Bellevue, Wash. This implementation includes both a run- 
time and a full version of ToolBook II. The full version of 
ToolBook II is only accessible by an author. The author then 
has the option of programming functions and enhancements 
beyond the capability of the VTC software system for 
incorporation into the training courses. 

Administration 

When an administrator logs on, the presented table of 
contents (FIG. 13) includes selections that allow the admin- 
istrator to review and maintain student records, generate 
reports of such things as user activity and VTC activity, and 
access utilities, including the master course list, log-in 
names and passwords, re-certification schedules and issu- 
ance of certificates of completion. 

The system includes a record for each student. The 
student record includes the date the student registered, the 
student activity log, and the student curriculum. The student 
records are stored on a separate computer so they cannot be 
edited by anyone except the administrator. To ensure integ- 
rity of the training records, the user activity information 
cannot be edited by anyone, including the administrator. The 
data can therefore be relied upon for certifying training for 
customer audits and to qualify for international 
certifications, such as ISO 9000. 

The administrator works together with the student to 
define an appropriate curriculum for the student. 

Various jobs require re -certification at regular time inter- 
vals. The system can generate a list of students requiring 
re-certification and manually or automatically, notify the 
students promptly. 

The administrator maintains a Master Course List of all 
the courses available. The VIP Directory is generated auto- 
matically from this List. 

Statistical information and graphs on the number of 
courses available over time, the average amount of time 
required to complete a course, the actual number of students 
taking and completing courses and other useful statistics are 
generated automatically from the student records. This infor- 
mation can be used by management to evaluate Return On 
Investment, help with budgeting and justify additional fund- 
ing of the training program. 
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Students 

After the student enters the Virtual Training Center and 
proceeds to a training workstation, the terminal displays his 
curriculum on the VIP Directory. (FIG. 14) It includes 

5 courses that the student has completed and courses to be 
completed. The student can only enter a course for which he 
has completed the prerequisites. A student can access 
courses only in the sequence specified by the administrator. 
As soon as a prerequisite course is completed, the next 

10 course becomes available for access automatically. Students 
are permitted to go back over completed courses for review 
or reference. 

The VIP Directory begins with a general level and 

15 expands to specifics as the student makes selections. For 
example, the top level may have a class of particular 
machines for which training modules exist. The second level 
may include machines of a certain type within this class, the 
third level may include different manufacturers of this type 

2Q of machines, the fourth level may include specific models 
systems of the selected machine and the fifth level may 
include training courses on basic theory of operation, 
operation, maintenance, etc. Before the student can select 
any particular module, he checks on the Directory to see 

25 whether he can take that course. 

Once a course is selected, a mechanical sound is heard 
resembling the sound of a real CD ROM being inserted into 
a drive and the CD slot of the VIP is shown to be loaded with 
a virtual CD ROM that carries the course title. This feature 

3 q is desirable not only to maintain the illusion of operating a 
real machine but to also indicate to the student the subject 
matter he is studying at all times. The CD ROM insertion is 
followed by the opening of the course's title page and from 
this point on the student is in control of the courses pace. 

35 Page navigation is accomplished with Forward and Reverse 
buttons. 

The VIP's digital clock is a multifunction device that 
automatically displays the time of the day, elapsed seconds 
during a review or the frame number of a digital video. The 

40 student can also use it as a stopwatch by just clicking on it 
with the mouse. 

The virtual monitor of the workstation has seven built-in 
pushbuttons that the student can use to fully control the 
digital video material included in a course. Thus a student 

45 can play, stop, pause, repeat, fast forward or single step 
forward or back any portion of a video procedure that needs 
to be examined or studied carefully. This level of control is 
very important in the learning process yet it is usually found 
only on very expensive professional video editing equip- 

50 ment. 

When a student completes a course, notification of the 
completion is sent to the administrator automatically. The 
curriculum is then updated either manually by the adminis- 
trator or automatically. After the curriculum is updated, the 

55 VIP Directory will indicate to the student that he can take 
courses for which the completed course is a prerequisite. 
Although automatic updating is easier, manual updating 
ensures that the administrator is aware of the progress that 
the students are making. Upon completion of a course, the 

60 system can print a completion certificate for presentation to 
the student by the training administrator, the student's 
supervisor, or other appropriate personnel. Also upon 
completion of a course, the student is allowed to access the 
material for that course in any order, rather than merely 

65 sequentially. The student can then use the course as a 
reference in his day-to-day performance of the task taught 
by the course. 
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To close the current course, a student merely presses the 
DIRECTORY button which retracts the CD ROM out of 
sight and returns to the VIP Directory. 

Workstation 

la a preferred embodiment, the system comprises the 
following hardware and software. FIG. 15 shows the rela- 
tionship between the hardware components of the system. 
The system is not limited to the hardware and software 
below, which are listed only as examples. 

System Software 

High performance Operating System (e.g., Microsoft 
Windows NT 4.0 Workstation) 
Hard Disk Defragmenter 

Authoring, administration and deployment software (Orizon 
Multimedia VTC 97) 

Application Software 
Authoring (e.g., Asymetrix ToolBook II) 
Instructional Design (Allen Communications Designer's 
Edge) 

Word processing, Spreadsheet, Database (Microsoft 
Office Professional) 

Vector/bitmap graphics, Flowchart, Animation (Graphics 
Suite) 

Non-Linear video editor 
Rae Sound wave editor 
OCR software 
Image capture and editing 

Computer System 
High performance CPU (Intel 200 MHz Dual PentiumPro) 
Adequate Random Access Memory (128 MB or more) 
A/V rated HDD 
Ultra Wide SCSI controller 
3.5" FDD 
4x CD Recorder 
SCSI CD ROM drives 
Full function Keyboard 
Pointing device 
High speed LAN adapter 

Video/Audio System 
1024x768 color monitor 
True Color (24 bit) video accelerator 
Editable, real time video capture at 30 fps near real time 
MPEG-1 compression 
Hi-8 VCR 
Hi -8 Camcorder 
Flatbed color scanner 
16-bit sound card 
Stereo headset with boom mike 

Console 

19" rack 

CD ROM drive bay 

Drawer for keyboard and pointing device 
Drawer for scanner 
Power: 110 VAC 

An embodiment of the present invention is suitable for 
use as a training standard, providing the advantage of 
allowing students to use a standard training setting for 
training on a variety of topics, thereby reducing the require- 
ment that students learn the use of different training tools 
and allowing the student to concentrate on the subject 
matter, rather than on the training equipment. 
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While a preferred embodiment of the present invention 
has been shown and described, it will be apparent to those 
skilled in the art that many changes and modifications may 
be made without departing from the invention in its broader 
aspects. The appended claims are therefore intended to cover 
all such changes and modifications as fall within the true 
spirit and scope of the invention. 

What is claimed is: 

1. A method of self •paced, computer-based training for 
individuals, the method usable in a distracting work envi- 
ronment by focusing the individual's attention onto the 
training, comprising: 

providing the student with a computer screen and an input 
device; 

providing on the computer screen a series of images of a 
student entering a virtual training center and moving 
through the virtual training center toward a virtual 
training workstation, the images of entering and mov- 
ing through the virtual training providing a virtual 
environment to focus the student away from the real 
environment and onto the training; 

providing on the computer screen an image of the virtual 
training workstation with the virtual training center in 
the background, the virtual training center image in the 
background providing the student a virtual training 
environment, thereby reducing distractions from the 
outside, non-virtual environment; 

presenting to the student on the virtual training worksta- 
tion a choice of training programs for presentation, the 
training programs presented including only those pro- 
grams that the student is qualified to undertake; 

presenting to the student on the virtual training worksta- 
tion an audio-visual presentation including first infor- 
mation to be learned; 

presenting to the student on the virtual training worksta- 
tion a test to determine whether the student has learned 
the first information; 

if the student has learned the first information, presenting 
to the student on the virtual training workstation second 
information; and 

if the student has not learned the first information, repeat- 
ing the presentation of the first information. 

2. The method of claim 1, further comprising automati- 
cally sending a message to a system administrator when the 
student learns a pre-defined set of information. 

3. The method of claim 1, in which providing on the 
computer screen an image of the virtual training workstation 
including providing lesson control features to the student, 
the lesson control features being presented as images of 
physical control devices, rather than as conventional com- 
puter control imagery. 

4. In a method of computer-based training in which a 
student is presented information at a student -controlled pace 
and in which the student is tested periodically and 
re-presented information that the testing shows was not 
learned, the improvement comprising providing the student 
a virtual training center environment metaphor to symboli- 
cally provide the student a virtual training environment 
separate from the working environment. 

5. The method of claim 4, in which providing the student 
a virtual training center environment includes presenting the 
student with an environment that simulates a physical, 
non-computer training environment. 

6. The method of claim 4, in which providing the student 
a virtual training center environment includes presenting the 
student with a portion of a computer screen that represents 
a physical training environment. 
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7. The method of claim 4, in which providing the student 
a virtual training center environment includes showing a 
virtual student entering a virtual training facility. 

8. The method of claim 5, in which the virtual training 
environment includes an isolated, serene environment. 5 

9. The method of claim 4, in which providing the student 
a virtual training center environment metaphor includes 
providing the student with lesson controls that simulate 
controls on physical devices, rather than computer-based 
controls, to minimizes computer-related imagery that can 10 
make students uncomfortable. 

10. In a computer-based training apparatus that presents 
information to a student at a student-controlled pace and that 
periodically tests the student and represents information that 
the testing shows was not learned, the improvement com- 15 
prising: 



320 

12 

means for providing the student a virtual training center 
environment metaphor to symbolically provide the 
student a virtual training environment separate from the 
working environment; and 
means for providing the student with lesson controls that 
simulate controls on physical devices, rather than 
computer-based controls, to minimizes computer- 
related imagery that can make students uncomfortable. 
11. The apparatus of claim 10, in which the means for 
providing the student a virtual training center environment 
metaphor includes means for providing the student with 
lesson controls that simulate controls on physical devices, 
rather than computer-based controls, to minimizes 
computer-related imagery that can make students uncom- 
fortable. 
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ABSTRACT 



A tutorial or diagnostic aid based on the representation and 
iterative interpretation of visual images is taught. A teaching 
or diagnostic session is created by overlaying an interpre- 
tation layer via a software application onto the visual image 
layer and synchronizing the two with the time code of the 
visual image. The interpretation layer allows the reviewer to 
identify image areas of interest by gesture and append 
comments thereto in real time; i.e. images or portions of 
images within the visual representation playback may be 
identified and labeled and have the concurrent commentary 
associated therewith. The comments are indexed and linked 
to a database of similar topics. The flow of the session is 
recorded to show the images, deictic gestures associated 
therewith, and commentary associated with the gestures, to 
enable subsequent users to playback a session and follow the 
flow of thought (i.e. image identification and commentary 
within the original session). Iterative sessions allow addi- 
tional image identification and commentary to be accom- 
plished. Additional database access through the index may 
further enhance a teaching session, or provide for research 
sessions and report generation. 

25 Claims, 5 Drawing Sheets 
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ITERATIVE VIDEO TEACHING AID WITH 
RECORDABLE COMMENTARY AND 
INDEXING 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a teaching or diagnostic 
aid and/or knowledge recording system whereby images or 
portions of images may be selected, and labeled, and con- 
current image-specific commentary applied to the visual 
image to create a teaching session which is indexable for 
later study, creation of further sessions, or research. 

2. Description of the Related Art 

In the diagnostic or teaching environment, such as a 
hospital or a medical school, great technological changes 
have taken place in terms of information storage, imaging, 
recording, and display technology. Yet the way in which 
knowledge is passed on has changed little. Data, or case, 
interpreters and learners meet in small groups and discuss 
individual case studies with a teacher or each other, to share 
knowledge and gain experience. When such sessions take 
place, the experiential knowledge is shared in informal and 
sometimes haphazard ways and only through a large number 
of repeated sessions is the individual's knowledge increased. 
Especially where the case study is in motion, e.g., a video 
presentation of biological events occurring in sequence over 
a time line, the record of such a teaching or diagnostic 
session is hard to establish with sufficient clarity in order to 
allow future users the benefit of a review of the session. 

Particularly in the field of internal medicine, imaging 
technology has revolutionized our understanding of abnor- 
malities in biological and neurological systems. Fast chang- 
ing technological improvements, impacting the quality of 
the images, make it even more difficult to compare cases and 
build an understanding of what the normal state is. The 
processes for combining data interpretation, documentation 
and retrieval are currently inefficient or lead to a high degree 
of abstraction in the diagnostic record. Despite all efforts in . 
automated image recognition, the human brain remains the 
best system for pattern recognition. Therefore, it would be 
desirable to match an image data base with appropriate 
indexing to the diagnostic abilities of the human data 
interpreter. 

By way of illustration, in the medical field, a cardiologist 
may review live or recorded echocardiograms with medical 
students on a video screen in an informal session as need or 
opportunity arises. Deictic gestures will usually include 
pointing to particular areas of the screen as the cardiac 
events happen in time. A commentary about the image will 
accompany each gesture. However, when the session ends 
the externalized knowledge expressed in gestures and 
accompanying commentary during the session is largely 
unrecorded and unavailable for future use. All that is left is 
fingerprints on the screen. 

Known diagnostic systems utilizing video recording and 
commentary are largely concerned with the image recording 
and display of the biological events to further automate 
diagnosis of the event and location of problem areas. Sys- 
tems such as detailed in U.S. Pat. No. 5,619,995 to Lobodz- 
inski or in U.S. Pat. No. 5,235,510 to Yaraada et al. are 
sophisticated tools of medical practice but do not appear to 
disclose critical aspects of a teaching, or interactive 
interpretation, system suitable for every day use in a real 
world, informal, teaching hospital environment. 

What is needed in such environments, and many other 
informal teaching or diagnostic environments, is a system 
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whereby all aspects of a teaching, or diagnostic, session 
occurring around an in- motion event, including gestures and 
commentaries of the students and teachers, may be recorded 
in time-synchronized fashion to capture the knowledge 

s expressed during the session for later use. It is desirable that 
such a system be easily learned and usable without special 
techniques. It is also desirable that the system be utilizable 
with a variety of video formats so as to be applicable to past 
and present recorded image data formats. It is further 

10 desirable that the knowledge expressed be cataloged, or 
indexed, for convenient retrieval and for comparison to 
contrasting, or similar, case studies to further heighten the 
efficiency of learning and sharing of experiential knowledge. 

15 DEFINITIONS 

"Frame" or "full frame" refers to an image covering the 
entire display screen. 
"Inserted frame" refers to a full image displayed as a 
2Q smaller portion inserted into the full frame analogous to a 
"window" or "picture in picture". 

"Subframe" refers to a portion of a fill image as selected 
by the viewer. 

"Concurrent commentary'* refers to comments made at 
25 the time of, or relating to, the subframe image selection. 

SUMMARY OF THE INVENTION 

The present invention teaches method and apparatus for 
recording experiential knowledge desirably including play- 
back of in-motion events (video), the recording of deictic 
gestures linked to particular images in the video, and the 
linking of commentary accompanying the deictic gestures, 
with the video images. The commentaries may then be 

35 indexed for easy retrieval and for cross-reference to images 
bearing on a similar topic. 

In general the present invention is accomplished by 
overlaying an interpretation layer on the video, or image 
layer, to record the deictic gestures of a viewer in time and 
location synchronization with the video, through a software 
application. The interpretation layer will also record com- 
ments and associate the comments with the gestures. Put in 
other words, the interpretation layer will create a gesture 
map and comment map overlaid on the video sequence with 

45 all three being synchronized to the video, whether by time 
code, frame and line code, or other image identifier accom- 
panying the video. The present invention is most efficacious 
with in-motion images but may work equally well with a 
series of static images as the image layer. 

50 Deictic gestures may be registered by any suitable means 
such as mouse operated cursor, a touch screen mechanism, 
light pen, or other means, now known or later developed, as 
dictated by the functional and economic considerations of 
the builder or user. Commentary may be continuously 

55 recorded and coded with the video and gestures, or the act 
of recording a gesture may open a window for comment 
specific to the gesture. Commentary may be entered vocally, 
through a keyboard, or through a menu of comment options, 
or any other means suitable to the application. The video 

60 may be paused, or looped between certain frames, to more 
easily accommodate the user in placing more than one 
gesture tag in a frame. The gestures and indexed commen- 
tary may be labeled a "session" and will be indexed appro- 
priately. Later sessions having additional gestures and com- 

65 mentary may be built upon a previous session. 

By replaying the session, a later viewer is privy to the 
deictic gestures and accompanying comments of previous 
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viewers thereby gaining a much more thorough understand- indicating to the viewer the objects of interest and commen- 

ing of the experiential knowledge, or areas of question, or tary therein. At this point the subframe 15 may again be 

both, expressed by the prior viewers. By linking through the selected by deictic touch, as at ref. no. 21, and additional 

index, the viewer may access any number of data bases to commentary added as at ref. no. 23 (FIG. 2C). Also, addi- 

enhancc their appreciation of the subject matter of the 5 tional subframes, e.g., 25, may be selected with attendant 

present session, further enhancing the value of the educa- comment windows 27 and comment entry. All new gestures 

tional experience. and commentary are recorded and desirably labeled as a 

separate session. AH commentary will desirably be indexed 

BRIEF DESCRIPTION OF THE DRAWINGS to aid in educational pursuit, or other decision making 

10 processes, as further explained below. 

FIGS. 1A-1D and 2A-2C illustrate teaching or diagnostic Referencing FIG. 3, exemplary apparatus for accomplish- 

sessions of the present invention. ing sessions according to the present invention will include 

FIG. 3 is a schematic representation of an apparatus a computer 29 for control of the video and session mainte- 

according to one embodiment of the present invention. nance; a display means 31, e.g., a monitor with a CRT 33 and 

FIG. 4 illustrates a video frame from a teaching session 15 a speaker 35, linked to the computer 29; and deictic gesture 

with tagged, or selected and labeled, subframes and their producing or capturing apparatus such as touch screen 

associated commentary. apparatus 37, a mouse 39, or keyboard 41, all linked to the 

HO. 5 illustrates a frame from a decision-support session 29 :, ^ «"° m611 ' ^ a PP ar4tus ^ ch » a 

wbere images of comparable cases are displayed simulla- microphone 43 or the keyboard 41 are necessary. The mouse 
neously with the case under study. 20 39 ™V »«" be ««« commentary from preselected 

n „ , .„ .„ e , , menus if the semantics of the discipline for which the 

FIG. 6 illustrates a frame from a research session where ^ j, mtended fof same 

unages from multiple cases are recalled by index top.es from ^ , & for ^ 

a data oase 

ing 45; the interpretation layer application 47 for operation 

DESCRIPTION OF PREFERRED 25 maintenance of the sessions; data storage 49, such as 

EMBODIMENTS ROM, disks, or the like; an indexing application 51 for the 

indexing of commentary or images selected within a session; 

A preferred embodiment will be described in the context and a database and attendant applications 53 for manage- 

of a medical teaching or diagnostic (hereinafter simply ment of broader teaching functions, such as comparing 

referred to as "teaching* 1 ) aid for the recording of echocar- 3Q sessions or searching images, as used to supplement indi- 

diogram review sessions. The ordinarily skilled artisan will vidual sessions. The specific construction of hardware and 

appreciate that the present invention is applicable to teach- software applications necessary to practice the teachings of 

ing or review sessions in many other fields such as the present invention are considered to be within the skill of 

criminology, history, or the like, and should not be taken as the person having ordinary skill in the art and therefore such 

limited to the discussed context, which is set forth by way 3S specifics are left to the discretion of the individual artisan, 

of example only. An antenna 55 is shown to schematically indicate that 

Referencing FIGS. 1A-1D, a brief portion of a teaching sessions may be remotely sent, accessed, or shared by 

session is shown to illustrate some principles of the present distance-learning techniques now known or later developed, 

invention. A full motion video event (hereinafter referred to to facilitate remote collaborative data interpretation ses- 

simply as "video"), such as an echocardiogram, is under- 40 sions. The present invention may also include communica- 

stood to be occurring on a monitoring system. An image of tion ports for interfacing with various networked data and/or 

interest to the image interpreter, e.g. a teacher or image management systems if desired, 

diagnostician, occurs on the echocardiogram in FIG. 1A. As indicated in FIG. 3, the video processor 45 will accept 

The teacher points to the image, and deictically gestures by input from a variety of sources 57 of live or recorded video 

touching at ref. no. 13 a particular area of the image 45 sources, including those transmitted from remote sources, 

designated here as a "subframc" 15 (FIG. IB). The touch is and will time code 59 or otherwise quantify the video layer 

sensed by a touch screen mechanism as known in the art and as necessary. Video may be reformatted, digitized, or the like 

a preselected area, or an area defined by the gesture, is within the scope of the present invention. The video layer 

highlighted. The motion of the video can be paused or time code 59 is shared with the interpretation layer 47 to 

looped if desired. A comment window 17 is then opened in 50 synchronize the two. The interpretation layer 47, which 

association with the selected, and concurrendy labeled or governs gesture and comment recordation, is overlayed on 

"tagged", subframc, allowing comment text 19 to be entered the video layer as indicated schematically by separate input 

concerning the selected subframe 15. If desired, the sub- lines 61, 63 for video layer and interpretation, respectively, 

frame 15 may be enlarged for further scrutiny, additional The overlay methodology is used to simplify the acceptance 

emphasis, or the like, as at FIG. ID. Motion of the echocar- 55 of multiple types of video source 57 and to protect the 

diogram video is then resumed, if it has been stopped, after original video source from any data corruption. A "session" 

comment has been entered under a command from the input line 65 is shown to schematically indicate capture and 

teacher. Each deictic gesture and synonymously labeled return of gestures and comments from the display means 31 

commentary are recorded in synchronization with the video to the computer 29. 

for later use. 60 Referencing FIG. 4, a frame from a teaching or diagnostic 
FIGS. 2A-2C illustrate playback of the first session session is shown. The image 11 is displayed in totality as an 
illustrated in FIGS. 1A-1D, and the ability of the system to inserted frame in one section of the video frame 12. Selected 
add additional comments to a previously tagged subframe, and labeled first, second and third subframe images or 
or add additional subframes and comments, thereby creating indications 67, 69, and 71 respectively, are overlaid by the 
another session. In FIG. 2A the replay of the recorded first 65 interpretation layer on top of the video frame 12. Acomment 
session is started, showing the image 11. At FIG. 2B the first box 73 associated and labeled synonymously with the sec- 
sub frame 15 and its associated comment has appeared ond sub-frame 69 displays the commentary created at the 
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time the subframe 69 was selected on the image display. A 
concept map 75 is displayed as a series of text within the 
video frame 12 to the left of the image 121. The text may be 
highlighted to indicate current status of the image display 
topic if desired. The concept map 75 may also be used as a 5 
menu to select commentary to be applied to selected sub- 
frames which can then be easily indexed according to the 
text by a database management application for later 
retrieval. Also, it is possible to build or modify the concept 
map architecture according to user input, such as the fre- 
quency of occurrence of the linking of certain concepts, 
inquiries, or the like. Automated generation of reports uti- 
lizing the diagnostic findings captured by a session accord- 
ing to the present invention is also envisioned. 

The concept map may also be used as a tool for calling up 15 
additional data associated with the image. Touch screen or 
cursor selection by a mouse are two exemplary methods of 
operating the concept map 75. The concept map 75 may also 
be linked to additional topics by a hierarchical tree, lattice, 
or semi-lattice of associated terminology for additional 20 
association, recordation, and indexing of a session. In alter- 
native embodiments, voice recognition may be used to 
capture and index natural language commentary with the 
associated deictic gestures and labeling applied during com- 
mentary. 25 

Further referencing FIG. 5, a video frame as used for a 
decision support mode of the system of the present invention 
is shown. The video frame 12 includes the image of interest 
11 in a large format inserted frame to the right of frame 12 
and displays a labeled subframc 15 and its associated 30 
comment window 17 as overlaid on the video image layer by 
the interpretation layer. To the left of the large format 
inserted frame image 11 are displayed multiple, e.g. first, 
second, and third, small format inserted frames 77, 79, and 
81 of similar images from different sessions to be used for 35 
comparison to the image of interest in the present session to 
arrive at a diagnostic conclusion. The small format images 
arc retrieved from the indexed database by keyword, image 
pattern recognition, or other known, or later developed 
means or applications. Image size selection and lay-out, 40 
such as two images occupying a split screen, may be varied 
at the control of the operator if desired. Recordation of 
additional deictic gestures and associated commentary may 
be added at any time to any part of the frame 12 by the 
interpretation layer. 45 

Referencing FIG. 6, a video frame as used in a pattern 
finding or research session mode of the system of the present 
invention is shown. Within the video frame 12, first through 
sixth multiple small format images from sessions of similar 
cases 83, 85, 87, 89, 91, and 93, respectively, are displayed, so 
The sessions will have been retrieved from the index data- 
base at the command of the operator. An overlay by the 
interpretation layer has placed a concept map text box 95, 
showing a semi- lattice of associated concepts, and an asso- 
ciated query text box 97, from the indexing protocol at the 55 
disposal of the operator to aid in the research function. Again 
all deictic gestures and associated comments may be entered 
and recorded by the interpretation layer during this mode of 
operation. Additional analytic applications such as statistical 
analysis programs, automated report production, or the like eo 
may be linked to the research session mode if desired. 

Having thus described a teaching system for the utiliza- 
tion of full motion video which captures the flow of thought 
and expression from live teaching sessions and indexes the 
same for increased dissemination and utilization of the 65 
knowledge expressed; it will be appreciated that many 
variations thereon will occur to the artisan upon an under- 



standing of the present invention, which is therefore to be 
limited only by the appended claims. 
I claim: 

1. A diagnostic and teaching aid including: 

a. means for displaying a sequence of images; 

b. means for selecting a portion of one image of the 
sequence of images via gestures of a diagnostician and 
means for recording a subframe image selection; 

c. means for recording a commentary of the diagnostician 
concerning a selected image portion; 

d. means for linking a recorded commentary concerning 
the selected image portion to the selected image por- 
tion; 

c. means for recording a plurality of image portion 
selections and a sequence in which the plurality of 
image portion selections were selected so as to create a 
teaching session; and 

f . means for indexing at least one of the recorded com- 
mentary and a selected image portion so as to make the 
teaching session retrievable through an index. 

2. The diagnostic and teaching aid of claim 1 wherein the 
means for displaying sequence of images includes a digital 
video processor. 

3. The diagnostic and teaching aid of claim 1 wherein the 
sequence of images are full motion images. 

4. The diagnostic and teaching aid of claim 1 wherein the 
means for selecting a portion of one image includes a mouse. 

3. The diagnostic and teaching aid of claim 1 wherein the 
means for selecting portion of one image includes a key- 
board. 

6. The diagnostic and teaching aid of claim 1 wherein the 
means for selecting portion of one image includes a touch 
screen. 

7. The diagnostic and teaching aid of claim 1 wherein the 
means for recording commentary includes a menu of termi- 
nology related to the images. 

8. The diagnostic and teaching aid of claim 7 further 
including means for associating related terms within the 
menu of terminology. 

9. The diagnostic and teaching aid of claim 8 further 
including means for recording a frequency with which terms 
are related by the diagnostician and adjusting the menu of 
terminology according to the frequency. 

10. The diagnostic and teaching aid of claim 1 further 
including means for indexing the selected image portion by 
automated image pattern recognition. 

11. The diagnostic and teaching aid according to claim 1 
further including: means for playing back the sequence of 
images while displaying the plurality of image portion 
selections and recorded commenta»y concerning the selected 
image portions, the play back being in the sequence in which 
the plurality of image portion selections were selected. 

12. A real time diagnostic and teaching aid including: 

a. means for displaying full motion images; 

b. means for selecting subframe images from displayed 
full motion images in real time via gestures of a 
diagnostician and means for recording a subframe 
selection; 

c. means for recording and linking concurrent commen- 
tary of the diagnostician to selected subframe images in 
real time; 

d. means for recording a sequence of subframe selection 
and linked commentary to create a teaching session; 

e. means for indexing the session by at least one of the 
commentaries and subframe images; 
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f. means for play back of the session; and 

g. means for adding additional commentary into the 
session. 

13. The real time diagnostic and teaching aid according to 
claim 12 further including means for selecting additional 5 
sub -frames during the session playback to create a further 
session; and means for indexing the further session. 

14. The real time diagnostic and teaching aid according to 
claim 12 further including a separate application Layer for 
overlaying image selection and concurrent commentary on 10 
the image data display without disturbing the image data. 

15. The real time diagnostic and teaching aid according to 
claim 12 further including means for displaying more than 
one session at a time as inserted frames. 

16. The real time diagnostic and teaching aid according to 15 
claim 12 wherein the means for indexing the further session 
includes means for indexing by at least one of the concurrent 
commentary, the additional commentary, and the subframes 
images. 

17. A diagnostic aid or teaching system comprising a 20 
computer having: 

a. a video layer for the display and quantization of a series 
of images; 

b. an interpretation layer, time synchronized to, and ^ 
running concurrently with, the video layer, and having 

a gesture recorder for identifying and recording 
selected portions of images in the video layer; 

c. the interpretation layer further including a comment 
recorder which records and displays concurrent com- 30 
mcnts and associates the concurrent comments with the 
gestures; 

d. synchronization means for synchronizing the video 
layer and the interpretation layer; 

e. the interpretation layer overlaid on the video layer so as 35 
to be displayed and run concurrently and in synchro- 
nization therewith; and 

f. an index manager for indexing of the comments or the 
select portions, or both, within a database; and 

40 

g. whereby a diagnostic or teaching session can be created 
comprising playback of the series of images concur- 
rently with image portions selected by gestures and the 
comments associated with the gestures. 

18. The diagnostic aid or teaching system of claim 17 45 
further comprising: storage means for recording activity 
within the interpretation application as synchronized to the 
video application. 

19. A method of producing a teaching session with 
in- motion video comprising: 
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a. displaying frames or inserted frames of full motion 
image data; 

b. selecting and recording subframe portions of the image 
data; 

c. appending comments to the selected subframe portions 
of the image data; 

steps "a", "b" and "c" making a session; 

d. recording the session; and 

e. indexing the session. 

20. The method of producing a teaching session with 
in-motion video of claim 19, further comprising: synchro- 
nizing the selecting and recording step and the appending 
step to the display of the full motion image data. 

21. The method of producing a teaching session with 
in-motion video of claim 20, further comprising: step a 
being performed in a separate application from steps b and 
c to prevent disturbing the full motion image data. 

22. The method of producing a teaching session with 
in-motion video of claim 20, further comprising: performing 
steps b) and c) during display of the full motion image data. 

23. The method of producing a teaching session with 
in-motion video of claim 20, further comprising: playing 
back the recorded session. 

24. The method of producing a teaching session with 
in-motion video of claim 20, further comprising: displaying 
more than one session at a time as inserted frames. 

25. A diagnostic and teaching aid method including the 
steps of: 

a. displaying a sequence of images; 

b. selecting a portion of one image of the sequence of 
images via gestures of a diagnostician; 

c. recording a commentary of the diagnostician concern- 
ing the image portion selection; 

d. linking a recorded commentary concerning an image 
portion selection to an image portion selection; 

e. recording a sequence of a plurality of image portion 
selections to create a teaching session; 

f. indexing at least one of the commentary and selected 
image portions so as to make the teaching session 
retrievable through an index; and 

g. playing back the sequence of images while displaying 
the plurality of image portion selections and commen- 
tary concerning the image portion selections, the play 
back including the sequence of image portion selection. 

» * * * * 
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1 2 

METHOD AND SYSTEM FOR PROVIDING Another system responding to questions is called Elisa. It 

INFORMATION IN RESPONSE TO responds to questions, and tries to emulate a psychiatrist. A 

QUESTIONS user enters a question into Elisa, which changes the entered 

question around to respond to the user. For example, the user 

This application is a continuation of U.S. patent appli- 5 enters, "I feel bad." Elisa might respond, "Why do you feel 

cation Ser. No. 09/347,184, filed on Jul. 2, 1999, the bad? " The system gets the user to talk, and presumably, the 

disclosure of which is hereby incorporated by reference, user feels better afterwards. The goal of the system is not to 

which is a continuation of U.S. patent application Ser. No. understand the user, but to encourage the user to commu- 

09/139,174, filed on Aug. 24, 1998, now U.S. Pat. No. nicate his problem. 

5,934,910, which is a continuation of U.S. patent application 10 m ^ syslems mal respond |Q questions written 

Ser. No. 08/758,896, filed on Dec. 2, 1996, now U.S. Pat. in computer languages. In such systems, the user 

No. 5,836,771; with the applications and patent being incor- reformulates his question into a program to access and to 

porated by reference into this application. process information & om a database. Someone not familiar 

with programming languages cannot get an answer from 
those systems. 

This invention relates generally to education and, more It should have been obvious that there is a need for a 

particularly, to computer learning based on question asking. method and a system that can teach a subject through 

Question asking is an important aspect in learning 

because we have a better understanding in a subject if we 20 
can ask questions. As opposed to passive leaning where we 

just absorb like a sponge, active learning based on asking invention is on a method and a system that can teach 

questions enhances understanding and helps us remember. a subject based on a user's questions. It is different from the 

However, if a person is learning from a computer system, he u^r learning a subject through passively absorbing the 

does not have the luxury of having a question-and-answer 25 materials. In this invention, he sets the learning pace, 

dialog with the computer. controls the learning process, and can learn by asking 

Asking questions not only focuses our attention on the questions, 

subject, it also fills gaps in our understanding. When we are In one embodiment, the system generates study materials 

learning from an instructor, typically we cannot comprehend 30 that introduce the subject to the user. After studying the 

everything. As our misunderstanding grows, very soon we presented materials, he can begin asking questions. The 

begin to lose track of the subject, and our interest in the system generates an answer to each question, and presents it 

subject wanes. Similarly, we lose interest in reading a book to him. The system also compares the question with one or 

with many individuals if we confuse their names. During more questions previously entered by him. The comparison 

those instances, asking questions to fill our gaps of misun- 35 determines his understanding level in the subject. If the 

derstanding might rekindle our interest in the subject or the comparison indicates that he is weak in a certain area, the 

book. system can present detailed study materials covering those 

A user's questions on a subject also indicate how much he arcas * Thc svstcm storcs me question he just asked, so 

understands the subject, If the user repeatedly asks questions 45 to l <> questions he might ask in thc future, 

in a certain area, he is weak in that area. 40 Typically the user does not ask one question and stop. He 

In view of the importance of question asking, many «»ya^ a scries of questions to understand me subject. After 

instructors include them in teaching. One of the most famous s y stem . has , ™P°l to hls questions, based on his 

teachers-Socrates-^ven used questions as his main tool to understanding level, the system may present to him addi- 

stimulate thinking and to teach. However, when a computer U ° n . al study matenals. The process may repeat with him 

teaches, the users cannot question the computer the same 45 askm S additional questions untd he understands the subject, 

way he can question his instructor. In another embodiment, the user can use the system to fill 

r 4 , , . • ■ n g-y gaps of misunderstanding in a subject. As he works on the 

Learning through a computer has its benefits. Computer u* . *u u <u . u * ,i_ . t_ 
„ * 1 *u- r? i cX subject through the computer, he encounters areas that he 

allows a user to learn at his own pace. For a class of thirty, • ■* , . , , . r . e „ ™. ... 

t . „ . 4 , .„ t K . . tl _ . . / ' „ does not understand or he has forgotten. This embodiment 

typically the instructor will not hold up the class just to 50 Questions in those areas 

clarify issues for one student. If students' levels of under- aU ° S dUn 10 get aDSWCrS ° D ^ uestIons m lDOse areas - 

standing are not the same, the instructor has to leave some ° tner aspects and advantages of this invention will 

of them behind. This problematic situation is prevalent in a become apparent from the following detailed description, 

classroom with students having different cultural back- which, when taken in conjunction with the accompanying 

grounds and non-uniform understanding levels. Computers 55 drawings, illustrates by way of example the principles of the 

can ameliorate such problems. If each student is taught by invention, 
his computer, he can control the computer so as to learn at 

his own pace. BRIEF DESCRIPTION OF THE DRAWINGS 

However, learning from a computer has its handicap. 6Q piG. 1 shows one embodiment of the present invention. 
When the student needs an answer for a question, problem 

arises because the computer cannot understand his question. FIG - 2 shows one embodiment of the answer generator in 

the present invention. 

There are computers responding to questions. One is the ^ T ^ 0 , A _ . , . 1 

system to locate books used in many libraries. Users can FIGS * 3A " B r s ^ ow P h y sical dements implementing one 

enter search-requests for books into the system. But such 65 embodiment of the present invention, 

systems are primitive as compared to those where a user can FIG. 4 illustrates a part of an hierarchy for the line-items 

learn a subject by asking questions. under fractions in the present invention. 
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FIG. 5 shows one embodiment implementing the present 
invention. 

FIG. 6 shows a set of steps to be used by the embodiment 
shown in FIG. 2. 

FIG. 7 shows a pre-defined context-free grammatical 
structure in the present invention. 

FIGS. 8A-B show examples of semantic rules applied to 
nouns in the present invention. 

FIG. 9 shows an example of semantic rules applied to a 
non-auxiliary verb in the present invention. 

FIG. 10 shows examples of semantic rules applied to 
adjectives in the present invention. 

FIG. U shows different approaches to resolve ambiguous 
question in the present invention. 

FIG. 12 shows another embodiment of the answer gen- 
erator in the present invention. 

FIG. 13 shows a set of steps to be used by the embodiment 
shown in FIG. 12. 

FIG. 14 shows a set of steps to fill gaps of misunder- 
standing in the present invention. 

Same numerals in FIGS. 1-14 are assigned to similar 
elements in all the figures. Embodiments of the invention are 
discussed below with reference to FIGS. 1-14. However, 
those skilled in the art will readily appreciate that the 
detailed description given herein with respect to these fig- 
ures is for explanatory purposes as the invention extends 
beyond these limited embodiments. 

DETAILED DESCRIPTION 

FIG. 1 shows one embodiment of a system SO in this 
invention. When a user starts working on a subject, a 
study-materials generator 52 generates introductory study 
materials using information from a database 106. A pre- 
senter 120 presents those materials. After studying the 
presented materials, the user might need to ask a question. 
He enters the question into the system 50 through an input 
device 56. An answer generator 100 retrieves the question 
and generates an answer based on information from the 
database 106 and a set of rules 64. The presenter 120 then 
presents the answer to him. A question comparator 60 also 
can compare the question with one or more questions he 
previously asked. Those questions were stored in the data- 
base 106. The comparison determines his understanding 
level in the subject. If the comparison indicates he is weak 
in a certain area, the study-materials generator 52 will 
retrieve study materials from the database 106 covering that 
area. The database 106 also stores the question just asked for 
future comparison. 

FIG. 2 shows one embodiment of the answer generator 
100 for answering a natural-language question, which is a 
question used in our everyday language. In the present 
invention, a question is defined as an inquiry demanding an 
answer; and an answer is defined as a statement satisfying 
the inquiry. 

An input device, such as a keyboard, a mouse or a voice 
recognition system, receives the natural-language question. 
Then a grammatical structure analyzer 102 analyzes the 
grammatical structure of the question for parsing the ques- 
tion into its grammatical components based on a pre-defined 
context-free grammatical structure. The analyzer 102 per- 
forms its tasks using a set of grammatical rules 104, and data 
from the database 106. Then a programming-steps generator 
108 automatically generates one or more instructions based 
on the components. The generator 108 performs its tasks 
using a set of semantic rules 110 and data from the database 
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106. The instructions flow to a programming-steps executor 
112, which executes the instructions. More than one set of 
instructions might be generated and executed. In at least one 
set of instructions, when it is executed, it queries and 
5 processes data from the database 106 for generating an 
answer to the question. The presenter 120, which is an 
output device, such as a monitor, a printer or a voice 
synthesizer, presents the answer to a user of the system. 

FIG. 3A shows one physical embodiment 150 implement - 
10 ing one embodiment of tie invention, preferably in software 
and hardware. The embodiment 150 includes a server com- 
puter 152 and a number of client computers, such as 154, 
which can be a personal computer. Each client computer 
communicates to the server computer 152 through a dedi- 
15 cated communication link, or a computer network 156. 

FIG. 3B shows one embodiment of a client computer 154. 
It typically includes a bus 159 connecting a number of 
components, such as a processing unit 160, a main memory 

20 162, an I/O controller 164, a peripheral controller 166, a 
graphics adapter 168, a circuit board 180 and a network 
interface adapter 170. The I/O controller 164 is connected to 
components, such as a hard disk drive 172 and a floppy disk 
drive 174. The peripheral controller 166 can be connected to 

25 one or more peripheral components, such as a keyboard 176 
and a mouse 182. The graphics adapter 168 can be connected 
to a monitor 178. Hie circuit board 180 can be coupled to 
audio signals 181; and the network interface adapter 170 can 
be connected to the network 120, which can be the Internet, 

30 an intranet, the Web or other forms of networks. The 
processing unit 160 can be an application specific chip. 

Different elements in the present invention may be io 
different physical components. For example, the input 
device 56, the presenter 120, the grammatical structure 

35 analyzer 102 and the grammatical rules may be in a client 
computer; and the study-materials generator 52, the question 
comparator 60, the database 106, the programming-steps 
generator 108 and the program executor 112 may reside in 
a server computer. In another embodiment, the database is in 

40 the server computer; and the input device 56, the study- 
materials generator 52, the question comparator 60, the 
grammatical structure analyzer 102, the programming-steps 
generator 108, the program executor 112 and the rules reside 
in a client computer. Yet in another embodiment, the 

45 embodiment 50 is in a client computer. 

In this invention, the subject can be broad or narrow. In 
one embodiment, the subject can cover mathematics or 
history, or it can cover the JAVA programming language. In 
another embodiment, the subject covers information in a car, 
50 such as a Toyota Canary, and the user wants to understand 
this merchandise before buying it. In yet another 
embodiment, the subject covers the real estate market in a 
certain geographical area, and again the user wants to 
understand the market before buying a house. 

As an example, the subject is American history. Historical 
facts and insights are arranged in chronological order. It 
starts with an introduction of the British empire before 1776, 
and then other information is arranged sequentially in time. 

60 In one embodiment, events happened within a certain time 
frame, such as one week, are grouped together as one item. 
And items can form a hierarchy structure. There can be a 
day-item, week-item, month-item and year-item. There can 
be long periods of time without significant events, and this 

65 leads to a month-item or a year-item. 

As another example, the subject is mathematics, which is 
separated into major-topics, minor-topics and line-items: 
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Major Topics Under Mathematics 

Calculus 

Geometry 

Trigonometry 

High School Algebra 
Minor Topics Under High School Algebra 
Decimal Numbers 
Polynomials 
Linear Equations 
Quadratic Equations 

Integers 
Line Items Under Integers 
Addition & Subtraction (Int+/-) 
Multiplication (lot*) 
Division 
Prime Numbers 
Factorization 
Common Divisor 

Fractions 

Line Items Under Fractions 

Addition and Subtraction with Common Denomina- 
tor (Frt+/- w/Comm Denom) 

Addition and Subtraction with Integers (Frt+/-, 
w/Int) 

Addition and Subtraction without Common Denomi- 
nator (Frt+/- w/o Comm Denom) 

Multiplication and Divisions with Integers (FrT ,/w/ 
lot) 

Multiplication and Division with traction (Fit*,/) 

Compound Fractions 

Fraction Reduction (Frt Reduction) 

Ratios and Proportions 
The major-topics, minor-topics and line-items form a hier- 
archy tree, which shows their relationships, and their relative 
difficulties. Each line-item also has a number of difficulty 
levels ranging from 1 to 10. FIG. 4 illustrates a part of such 
an hierarchy for the line-items under fractions. For every 
path in the figure, there is a number and an alphabet in 
parenthesis. The number denotes the minimal difficulty 
level, and the alphabet denotes the minimal grade that the 
user has to achieve before the user is qualified to work on the 
subsequent line-item. For example, the user has to achieve 
at least a difficulty level of 8 and a grade of B for the 
line-item of Int+/- before the user is qualified to work on the 
line-item of Frt+/- w/Comm Denom. Generating study 
materials for each difficulty level of each line-item should be 
obvious to those skilled in the art. 

In one embodiment, the system 50 further includes an 
access gate 62. When the user wants to learn a subject, he 
enters his name and may be his password with the title of the 
subject he wants to learn through the input device 56 into the 
access gate 62. The access gate 62 accesses the database to 
determine if he has used the system before, or if the user has 
used the system to learn the subject before. If he has not used 
the system to learn the subject before, the access gate 62 asks 
the study-materials generator 52 to retrieve introductory 
study materials on the subject for the user. In another 
embodiment, the subject does not have any introductory 
materials, and he starts the learning process by entering 
questions. 

In yet another embodiment, the database 106 stores the 
questions asked by a number of prior users, and the question 
comparator 60 compares the questions asked by them to 
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determine questions that are commonly-asked. Comparison 
processes will be described below. The term "cornnonly- 
asked" may be defined as being asked by more than 50% of 
the prior users, or by other metrics. The study-materials 
5 generator 52 retrieves a set of study materials answering the 
commonly-asked introductory question, and presents them 
to him. Answer-generation processes will be described 
below. 

For different parts of the subject, again there might be one 
10 or more questions commonly asked by others. Answers to 
those questions can be presented to him when he starts 
working on those areas of the subject. 

After learning the introductory material, the user may start 
asking questions by entering them into the system. Each 
15 question may be entered into the system 50 orally through a 
voice recognition input device, or through a keyboard, or 
other types of input device 56. FIG. 5 shows one embodi- 
ment 190 implementing the present invention. First, the 
answer generator 100 retrieves (Step 192) the question 
2 q entered, and generates (Step 194) an answer to the question. 
The presenter 120 then presents (Step 196) the answer to the 
question, and the comparator 60 compares (Step 198) with 
one or more questions previously entered by the user to 
determine his understanding level in the subject. The 
25 sequence of the steps of comparing (198) and presenting 
(196) can be changed, or the two steps can be simulta- 
neously performed. 

In one embodiment, the question just asked by the user is 
stored in the database 106 with his identity. In another 
30 embodiment, the database also stores a time-stamp indicat- 
ing the time when the user asks the question. 

There are a number of ways to generate (Step 194) an 
answer to the question entered. The following description 
starts with answering natural-language questions that are 
3 5 grammatically context-free, and then extends to answering 
other types of questions. 

A natural-language question can be in English or other 
languages, such as French. Examples of natural-language 
questions are: 
40 Who is the first President? 
What are the Bills of Right? 
Where is the capital of Texas? 
What is the immediate cause to the Civil War? 
Why did President Nixon resign? 
45 Who is the third President? 

Who is the President after John Kennedy? 
When did President Lyndon Johnson die? 
When was President Nixon born? 
50 What is the derivative of sin(x+4) with respect to? 
Why is delta used in step 4 of the proof? 
A statement that is not based on a natural language is a 
statement that is not commonly used in our everyday lan- 
guage. Examples are: 
55 For Key in Key-Of(Table) do 
Do while x>2 

A grammatically-context-t-free question is a question 
whose grammar does not depend on the context. Each word 
in the question has its own grammatical meaning, and does 

60 not need other words to define its grammatical meaning. 
Hence, the grammatical structure of the question does not 
depend on its context. Note that "a word" can include "a 
number of contiguous words/' This is for situations where a 
term includes more than one word but has only one gram- 

65 matical meaning, such as the preposition "with respect to." 
The question includes one or more grammatical compo- 
nents. A grammatical component is a component with one or 
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more grammatical meanings, which are defined by a set of 
grammatical rules to be explained below. For example, the 
word "president" is a noun, which has a grammatical mean- 
ing. So the word "president" is a grammatical component. 

The present invention includes a database, which can be 
a relational database, an object database or other forms of 
database The database can reside in a storage medium in a 
client computer, or a server computer, or with part of it in the 
client computer and another part in the server computer. 

In one embodiment, the database includes a number of 
tables. A tabic can be treated as a set of information or data 
grouped together that have some common characteristics. 
The data in each table can be further divided into different 
areas, and each area is represented by an attribute, which is 
equivalent to an identifier for a group of data that are more 
narrowly focused than all the data in a table. Id the present 
invention, tables and attributes have similar function, except 
a tabic may be considered to have a broader coverage, and 
an attribute a narrower focus. In some examples, a table has 
two dimensions, as will be explained below. 

Some values or data in the database may be unique. For 
example, if a value is a person's social security number, that 
value is unique. Such values are known as key values, and 
their corresponding attributes are known as key attributes. 
Note that a table can have one or more key attributes, and a 
key attribute may in turn be formed by more than one 
attribute. 

One embodiment of the database 106 includes a gram- 
matical table 114, one or more topic- related tables 116, and 
two semantic tables, 118 A and USB. In a general sense, the 
grammatical table 114 determines the grammatical meaning 
of each word in the question, such as whether a word is a 30 
noun or a verb. Each topic-related table 116 groups data 
related to a topic together in a specific format. Separated into 
a topic-dependent semantic table USA and a topic- 
independent semantic table U8B, the semantic tables define 
the semantic meaning of each word, such as whether a word 
refers to an algorithm or data in a topic-related table. 

The grammatical table 114 defines the grammatical mean- 
ings of words used in the natural-language question. If 
questions entered into the system is limited to only one 
subject, such as history, the grammatical table will include 
words in that subject, and words commonly-used by a user 40 
of the system in asking questions. Each word in the table 
may be defined in the following format: 



CREATE TABLE Grammatical ( 



word 



grammatical- meaning 



) 



Character string NOT NULL, 
//the word 

Character string NOT NULL, //e.g. 
"Examiner" 

//has "noun" as its grammatical 
meaning 



10 ) 



15 



20 



25 



35 



50 



Each topic-related table combines data related to a topic 
in a specific format. As an example, one table includes all the 
data related to the Presidents of the United States, and 
another includes all the data related to the First Ladies of the 
United States. The table may be two-dimensional, and 
include a number of columns and rows. All the data in a 
column or a row typically have one or more common 
characteristics. For example, one row includes data that 
identify all the bills passed by the Presidents. For a two- 
dimensional table, data in a row can have one characteristic, 
and data in a column can have another characteristic. For 
example, data in one column identify the heights of the 
Presidents, and data in a row identify data related to one 
specific President; the following describes an example of 
data alone the row: 



55 



CREATE TABLE PRESIDENT AS ( 

//Table of all U.S. Presidents & Vice Presidents 

Name Character string KEY, //President Name-a key attribute 

Position Character string, //President, Vice President 

Start_Year Integer, 

End_Year Integer, 

Bora_Date Date, 

Death_Date Date, 



//First Year of Presidency 
//Last Year of Presidency 
//Date of Birth 
//Date of death 



There is also a table-structure dictionary, which defines bow 
the topic-related tables arrange their data. This dictionary is 
typically not considered as a part of the database. It does not 
contain topic-related data, but it contains structures of the 
topic-related tables in the database. Many database manage- 
ment systems automatically generate the table-structure dic- 
tionary based on the programing statements defining the 
topic-related tables, such as the CREATE clauses in SQL- 
like languages. As an example, the table-structure dictionary 
defines the structure of the data in the above President table 
by indicating that the first entry represents the name of the 
president, the second the position, and so on. Thus, the 
dictionary can contain the name of the table (the table 
name), the name of the table's attributes (attribute names), 
and their corresponding data types. 

A word in the question may need one or both of the 
semantic tables. The topic-independent semantic table 118B 
defines whether a word stands for an algorithm or data in a 
topic-related table. Such a table may be defined as follows: 



CREATE TABLE Topic_Independent_Scmantic ( 
word NOT NULL, //the word 

semantics, //Indicates if the word refers to data in a 

//topic-related table, an algorithm etc. 
If the 

//word is mapped to an algorithm, that 
//algorithm will also be identified, 
as will be 

//further explained below, 
synonym, //A word might have synonyms, 

as will be 

//further explained below. 



) 



45 



Words with similar meaning are grouped together and are 
represented by one of those words as the synonym for that 
group of words. If a word does not have other words with 
similar meaning, the synonym is the word itself. 

Many words do not point to an algorithm. They corre- 
spond to data in topic-related tables. The topic-dependent 
semantic table 118A identifies the semantic meaning of 
those words through matching them to data in topic-related 
tables. For example, Ihe adjective "first" applying to the 
President's table may operate on the data under the inaugu- 
ration date attribute; on the other hand, the adjective "first" 
applying to the First Ladies* table may operate on the data 
under the date of death attribute. Such a topic-dependent 
table 118A may be defined as follows: 



60 



CREATE TABLE Topic_Dependent_Semantic ( 



65 



Table„Name NOT NULL, 

Who_Attribute, 
When_Attribute, 



//For a table with the name 
Table_Name: 

//The attribute associated with *who' 
//The attribute name associated with 
'when' 
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-continued 

{i-pnonoun} Attribute, //The attribute associated with the 

//{i-pronoun}. The symbols { } 
denote the 5 
//word it contains. Here, the word is 
//an i-pronoun. 

{Adj}_Attn"bute, 

//The attribute associated with the adjective {adj}. In this 
//example, the word is an adjective. 10 

{Noun}_AUribute, 

//Attribute name associated with the noun {noun}. Certain 
//nouns may refer instead to an algorithm, such as "sum." 

) 



In one embodiment, the grammatical analyzer 102, the 
grammatical rules 104 and the grammatical table 114 are in 
a client computer. The programming-steps generator 108, 
the semantic rules 110, the semantic tables 118 and the 
table-structure dictionary are in a middleware apparatus, 
which can be a Web server. The programming-steps executor 20 
112 with the topic-related tables are in a back-end server, 
which can be a database server. 

One embodiment includes a computer-readable medium 
that encodes with a data structure including the semantic 
tables 118. Another embodiment includes a computer- 25 
readable medium that encodes with a data structure includ- 
ing the semantic tables 118 and topic-related tables 116. Yet 
another embodiment includes a computer-readable medium 
that encodes with a data structure including the semantic 
tables 118 and the grammatical table 114. Yet a further 30 
embodiment includes a computer-readable medium that 
encodes with a data structure including the grammatical 
table 114, the topic-related tables 116 and the semantic 
tables 118. 

FIG. 6 shows a set 200 of steps to implement one 35 
embodiment of the present invention. A natural-language 
question is entered into the answer generator 100. The 
system analyzes (Step 202) the grammatical structure of the 
question so as to parse it into its grammatical components, 
based on a pre-defined context-free grammatical structure. 
This task uses a set of grammatical rules 104 and the 40 
grammatical table 114. Then, the system transforms (Step 
206) at least one component into one or more instructions 
using a set of semantic rules 110 with one or both of the 
semantic tables 118. Then, the one or more steps are 
executed (Step 208) to access and process data from one or 45 
more topic-related tables so as to generate an answer to the 
question. 

In another embodiment, the programming-steps generator 
108 transforms alt the grammatical components of the 
question into instructions using semantic rules 110 with one 50 
or both of the semantic tables. Then the executor 112 
executes all the steps to access and process data from one or 
more topic-related tables for generating an answer to the 
question. 

Grammatical Structure Analyzer 55 

In one embodiment, the analyzer 102 scans the question 
to extract each word in the question. Then the analyzer 102 
maps each extracted word to the grammatical table 114 for 
identifying its grammatical meaning. For example, the word 
"Clinton" is identified by the grammatical table to be a 60 
proper noun; and the word "sum" is a noun. After estab- 
lishing the grammatical meaning of each word, the analyzer 
102 uses a set of grammatical rules to establish the gram- 
matical components of the question based on the pre-defined 
context-free grammatical structure. 65 

For a number of words, their grammatical meaning 
depend on their adjacent words. In one embodiment, the 
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analyzer 102 combines each word with its contiguous words 
to determine its grammatical component. For example, if the 
word is "With," in analyzing its Grammatical meaning, the 
analyzer 102 identifies its contiguous words. If its contigu- 
ous words are "respect to,** then the three words are com- 
bined together and are considered as one preposition. Thus, 
to determine grammatical meaning of a word, the analyzer 
identifies that word, and then a number of words following 
it, such as two words following it. The analyzer 102 analyzes 
the identified words as a unit. If the analyzer 102 cannot 
identify the grammatical meaning of that sequence of words, 
the analyzer 102 removes the last word from the sequence, 
and analyzes them again. The process repeats until either a 
grammatical meaning is found, or there is no more word. 
Any time when the analyzer 106 has identified a grammati- 
cal meaning, that word or sequence of words would be 
considered as one unit. 

In one embodiment, the pre-defined context-free gram- 
matical structure is shown in FIG. 7 and is as follows: 

<Question>°<i-pronounxaux-verbxnoun-phrase> 
[<verb-phrase>] 

where: the symbols <> denote whatever inside is a meta- 
symbol, which has a grammatical meaning; the meta- 
symbol is not in the grammatical table. 

The symbols [] denote whatever inside the bracket is 
optional. 

<I-pronoun> denotes an interrogative pronoun which is a 
pronoun used in asking questions, and can be one of the 
following: what, when, where, who, whom, whose, 
which, and why. 
<Aux-vcrb> denotes an auxiliary verb, and can be any 

form of the verb "to be," or "do." 
<Noun-phrase> is defined as <group-of-nouns> 
[<prepositional-noun-phrase>] 
where: <group-of-nouns> is defined as: 
[<modify-article>] <adjective> *<one-or-more-nouns>; 
the symbol * denotes zero or more; 
<modify-article> is defined as a modified article, 

including a, an, the, this, these and those; and 
<one-or-more-nouns> denotes one or more nouns; 
and 

<prepositional-noun-phrase> is defined as a 
<preposition><noun-phrase>. 

< Verb-phrase > denotes a non-aux-verb, and is defined as 
<non-aux-verb>[<prepositional-noun-phrase>]. 

<Preposition> denotes a preposition defined in the gram- 
matical table. 

<Non-aux-verb> denotes a verb defined in the grammati- 
cal table and is not an <aux-verb>. 

<Noun> denotes a noun defined in the grammatical table. 

<Adjective> denotes an adjective defined in the gram- 
matical table. 

The pre-defined structure is only one example to illustrate 
the present invention. Other context-free grammatical struc- 
tures are applicable also. Generating different context-free 
grammatical structures should be obvious to those skilled in 
the art. 

In the present invention, a word or a set of words that can 
fit into the structure of a meta-symbol is a grammatical 
component. For example, the phrase "with respect to x" is a 
grammatical component, whose grammatical meaning is a 
prepositional-noun-phrase. 

In the present invention, grammatical rules and the pre- 
defined grammatical structures are linked. Once the rules are 
set, the structures are determined. Similarly, once the struc- 
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tii res are determined, a set of rules can be found. For 
example, based on the pre-defined structures, one grammati- 
cal rule is that "a group-of-nouns preceding a prepositional- 
noun-phrase is a noun-phrase." 

The grammatical table defines the grammatical meaning 
of each word. In one embodiment, the table is a part of the 
grammatical rules. In another embodiment, all the gram- 
matical rules that define the grammatical meaning of each 
word are separated from the rest of the grammatical rules, 
and are grouped together to establish the grammatical table 
114. 

A number of examples on analyzing a question for parsing 
it into its grammatical components based on the pre-defined 
grammatical structure are: 

1. What is the derivative of sin(x+4) with respect to x? 
Starting from the right hand side, 

(x) is a noun 

so (x) is a group-of-nouns 
so (x) is a noun-phrase 

so (with respect to x) is a prepositional-noun-phrase 

(sin(x+4)) is a noun 

so (sin(x+4)) is a group-of-nouns 

so (sin(x+4) with respect to x) is a <group-of- 
nounsxprepositional-noun-phrase> 

so (sin(x+4) with respect to x) is a noun-phrase 

so (of sin(x+4) with respect to x) is a prepositional- 
noun-phrase 

(derivative) is a noun 

(the) is a modify-articlc 

so (the derivative) is a group-of-nouns so (the deriva- 
tive of sin(x+4) with respect to x) is a <group-of- 
nounsxprepositional-noun-phrase> 

(is) is an aux-verb 

(What) is an i-pronoun 
Thus, the question is of the structure 

<i-pronounxaux-verbxgroup-of- 
nounsxprepositional-noun-phrase>. 

2. Why is delta used in step 4 of the proof? 
Starting from the right: 

(proof) is a noun 

(the) is a modify-article 

so (the proof) is a group-of-nouns 

so (the proof) is a noun-phrase 

so (of the proof) is a prepositional-noun-phrase 

(4) is a noun 

(step) is a noun 

so (step 4) is a group-of-nouns 

so (step 4 of the proof is a <group-of- 
nouns><prepositional-noun-phrase> 

so (step 4 of the proof) is s noun-phrase 

so (in step 4 of the proof) is a prepositional-noun- 
phrase 

(used) is a verb as defined by the grammatical table 
so (used in step 4 of the proof) is a verb-phrase 
(delta) is a noun 
so (delta) is a group-of-nouns 
so (delta) is a noun-phrase 
(is) is an aux-verb 
(Why) is an i-pronoun 
Thus, again the question is of the structure 

<i-pronoun><aux-verb><noun-phrasexverb-phrase>. 

3. Why did President Nixon resign? 
Starting from the right-hand side 

(resign) is a verb 

so (resign) is a verb-phrase 

(Nixon) is a noun 
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(President) is a noun 

(President Nixon) is a one-or-more-nouns 
so (President Nixon) is a noun-phrase 
(did) is an aux-verb 
5 (Why) is an i-pronoun 

Thus, the question is of the structure 

<i-pronoun><aux-verbxnoun-phrase><verb-phrase> 
Many questions cannot be parsed based on the pre-defined 
context-free grammatical structure. In this disclosure, these 
10 questions are considered as ambiguous questions, and still 
be analyzed through methods explained later in this disclo- 
sure. If there are more than one such pre-defined context- 
free grammatical structure stored in the system, the question 
entered will be parsed based on each structure individually. 
The question only has to be successfully parsed based on one 
15 such structure. If the question cannot be parsed based on all 
the pre-defined context-free grammatical structures, the 
question will be considered as an ambiguous question. 
Programming-steps Generator 
The programming-steps generator 108 transforms at least 
20 one grammatical component of the question using a set of 
semantic rules and one or both of the semantic table to 
generate a set of instructions. The semantic rules and the 
semantic tables depend on the pre-defined context-free 
grammatical structure, which the parsing process bases on. 
25 In one embodiment, the semantic rules are also embedded in 
the semantic tables. In a general sense, the generator 108 
directs different grammatical components in the question to 
algorithms or to data in the topic-related tables. 
To help explain the present invention, a number of func- 
30 dons are created as shown in the following: 
Keys-Ofi(Table) 

This function extracts all the key attributes in the 
identified table. 
3J Attributes-Of^Table) 

This function extracts all the attribute names in the 
identified table. 
Attribute-Names({ adjective}, Table) 
This function identifies one or more attributes when the 
40 {adjective} is applied to the table. 

Attribute-Names({noun}, Table) 

This function identifies one or more attributes when the 
{noun} is applied to the table. 
Attribute-Name({i-pronoun}, Table) 
45 This function identifies the attribute vhen the 
{i-pronoun} is applied to the table. 
Tables-Of( {proper noun}) 

This function identifies one or more tables that contain 
the {proper noun} as a key value. It can be derived 
50 by the following program: 

T-Nameso""; 

for Table in {all Tables}//{all Tables} is a list of 
topic-related tables 

do 

55 for Key in Keys-Of(Table) 

do 

if any value of the attribute Key in the Table 
contains {proper noun} 
then T-N Names»T-Names+Table 
60 endif 
endfor 
endfor 

return T-Names 
Synonym({word}) 
65 This function identifies the synonym corresponding to 
the word. The synonym can be found in the topic- 
independent-semantic table. 
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Methods to create the above functions should be obvious 
to those skilled in the art of programming. 

Based on a number of semantic rules, the programming- 
steps generator 108 generates instructions based oo the 
grammatical components in the question. The following 
shows examples of different instructions generated to illus- 
trate the present inventions. The instructions generated are 
either in a SQL-like, a LISP-like or a C-like language though 
other programming languages are equally applicable. 

A Proper Noun 

Agrammatical component in the question can be a proper 
noun, which implies that it has a grammatical meaning of a 
proper noun. One set of semantic rules is that the 
programming-steps generator 103 transforms the proper 
noun into instructions to select one or more topic-related 
tables, and then transforms other grammatical components 
in the question into instructions to select and to operate on 
data in the tables for answering the question. 

Using the topic-dependent semantic table 118A, the 
programming-steps generator 108 first retrieves all tables 
where the proper noun is an attribute. Then, as shown in the 
topic-dependent semantic table, all key attributes in those 
tables are identified, and each of them is matched to the 
proper noun. The table of any key attribute that matches the 
proper noun is selected for additional operation by the 
remaining grammatical components in the question. 

A proper noun may consist of more than one word, such 
as the "Bills of Right." A proper noun can be a lower-case 
word, such as "moon." 

In one example, the corresponding instructions are as 
follows: 

for Table in Table-Of( {proper noun}) 
do 

for Key in Kcys-Of(Table) 
do 

xKSELECT . . . 
FROM Table 

WHERE Key MATCH {proper noun}) 

// The above clause has the meaning of "where 

the key attribute 
// in the table matches the proper noun." 
if x is valid then done 

// if the SELECT function successfully identifies one 

or more attributes, 
// x is valid, 
endfor 
endfor. 

Common Nouns 

One grammatical component in the question can be a 
common noun. The programming-steps generator 108 might 
transform the common noun into instructions to select a 
topic-related table, an attribute name, a synonym of an 
attribute name, the data under an attribute, or an algorithm. 

As shown in FIG. 8A, if the noun is the name of a 
topic-related table as shown by the topic-dependent seman- 
tic table 118A, then the programming-steps generator trans- 
forms the noun into instructions to select (Step 252) that 
topic-related table, and transforms (Step 254) other compo- 
nents in the question to identify data in the table and to 
operate on them, if necessary. 

If the noun denotes an attribute name or a synonym of an 
attribute name, again as shown by the topic-dependent 
semantic table 118A, the programming-steps generator 
searches and identifies the attribute based on the noun. The 
instruction generated can be, for example, modifying a 
SELECT clause as follows 
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for Attribute in Attribute-Names({noun}, Table) 
do 

SELECT Attribute from Table 
5 endfor 

After all of the relevant attributes have been identified, data 
in them are retrieved for further processing by other parts of 
the question to generate an answer. 
If the noun denotes the data under an attribute, the 

10 programming-steps generator identifies the data, with its 
corresponding attribute and table. The instructions generated 
can be, for example, (1) identifying each table in the 
function Tables-Of({noun}); (2) for each table identified, the 
function Attribute-Names({noun}, Table) returns the cone- 

15 sponding attributes containing the {noun} in that table; and 
(3) the remaining parts of the question operate on informa- 
tion under each attribute to generate the answer to the 
question. One set of instructions achieving such objectives 
is as follows. 

20 for Table in Tables-Of({noun}) 
do 

for Attribute in Attribute-Names({noun}, Table) 
do 

2S SELECT. . . 

FROM Table 

WHERE Attribute={ooun} 
endfor 

30 

endfor 

As shown in FIG. 8B, the programming-steps generator 
might identify the algorithm (Step 262) corresponding to the 
noun; the algorithm is then applied (Step 264) to data 

35 selected by grammatical components in the question other 
than the common noun. For example the noun "sum" 
indicates accumulating results; the noun "count" indicates 
computing the cardinally of the results; and the noun "prod- 
uct" in mathematics indicates multiplying results. The topic- 

40 independent semantic table 118B can point to locations to 
get the algorithm. 
A Group of Nouns 

If the question includes a group of nouns linked together, 
such as XI X2 X3 . . . Xn, then XI to Xn-1 can modify, the 

45 final noun Xn, which is known as the primary noun. In other 
words, the programming-steps generator operates on the 
primary noun as a common noun, or a proper noun, which- 
ever it may be, and the remaining nouns XI to Xn-1 further 
operate on dataAable(s) selected by the primary noun. 

so Non-Auxiliary Verbs 

One grammatical component can be a non-auxiliary verb. 
It relates to one or more events or an action, which has a 
number of attributes; and it might have words with similar 
meaning. One approach is to identify the verbs with similar 

55 meaning. Then other components in the question identify 
data in the attributes of the identified verbs for answering the 
question. 

A verb can be related to many different events. As an 
example, the verb is "nominate": one event can be President 
60 Bush being nominated to be the President, and another event 
can be President Clinton being nominated to be the Presi- 
dent. 

However, an event is related to a verb. The attributes of 
the event can have a subject-agent, which is the agent 
65 performing the event, such as the party nominating the 
president. Typically, the preceding noun phrase before the 
verb identifies the subject-agent. The event can have an 
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object-agent if the verb is a transitive verb, which is the 
agent acted upon by the event, such as the president being 
nominated. 

Each event has a duration that is between a starting and an 
ending time. For example, if the event is "walk," its duration 
starts with the sole of a foot changing its position from 
touching the ground to not touching the ground, and then 
ends with the sole back to touching the ground again. 

Non-auxiliary verbs are grouped together in an event 
table, which is a topic-related table, with the topic being 
events. The following is an example of an event in the table: 



CREATE TABLE EVENT ( 

Verb_word Character String NOT NULL, 

//The verb that associates with the event 
Subjcct_Ageat Character //Agent name performing the event 
String, 

ObjecL^Agcnt Character //Agent name acted upon by the 

String, //event 
Start_Time Time, //Starting time of event 
End„Timc Time, //Ending time of event 
Description Character //Describes the event 

Suing, 

Keytd Integer, //Unique number identifying the 

event 



) 



The subject-agent, object_agent etc. arc attributes related 
to the verb_word, which is associated with an event. 

There might be non-auxiliary verbs with similar meaning 
as the non-auxiliary verb in the question. These verbs can be 
identified by the synonym in the topic-independent semantic 
table. As an example, the verbs of breathe and inhale have 
similar meaning. 

As shown in FIG. 9, the programming-stcps generator 108 
transforms the non-auxiliary verb in the question into one or 
more instructions, which select (Step 300) one or more verbs 
with their attributes in the event table. The one or more verbs 
have similar meaning as the non-auxiliary verb. Then other 
components in the question identify data (Step 302) in he 
attributes for answering the question. The selected verbs can 
be put into a temporary table or a view (a database 
terminology) as follows. 

CREATE VIEW Vcrb_View({verb}) As 
// View is a logical table that is created only when it is 
needed. 

// All events matching {verb} are grouped from the 
event table 

// to form the view. 

SELECT* FROM EVENT 
// here * denotes all of the attributes 

WHERE Synonym({verb})=Verb_word; 
The attributes of the selected verbs are also identified. Then, 
the programming-steps generator 108 generates additional 
instructions based on other components in the question to 
identify data (Step 302) in the selected attributes for answer- 
ing the question. 

Events might be related. Two events may form a sequen- 
tial relationship, where one event follows another event, 
such as eat and drink. Two events may form a consequential 
relationship, such as braking and stopping, with the braking 
event causing the stopping event. Many small events may 
make up a big event, with the big event containing the small 
events; this leads to containment relationships. Also, events 
may be related because they involve the same subject-agent; 
and events may be related because they involve the same 
object-agent. 

An event-relationship table describes relationships among 
events. It can have the following format: 



CREATE TABLE EVENT_RELATIONSHIP ( 

Key Id 1 Integer, //Keyld of an event 

Keyld2 Integer, //Kcyld of another event 

Relationship Character String, 

//Relationship, such as sequential, consequential, containment 

etc. 



) 
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Interrogative Pronouns 

Based on the interrogative pronoun in the question, the 
programming-steps generator 108 Generates one or more 
instructions to select one or more attributes in one or more 
tables. Those tables have been selected by grammatical 
components in the question other than the interrogative 
pronoun. The function Attribute-Name({i-pronoun}, Table) 

?e Derates the attribute name corresponding to the 
i-pronoun} . 

One way to generate a SQL-like instruction correspond- 
ing to the {i-pronoun} is to modify a SELECT clause: 
SELECT Attribute-Name({i-pronoun}, Table) FROM 

Table 
Determiners 

Examples of a set of semantic rules on determiners are: 
If the determiner is "a" or "an", select any result from the 

previous query. 
If the determiner is "some", select more than one result 

from the previous query. If the previous query yields 

only one result, that result will be selected 
If the determiner is "all", select all result from the 

previous query. 
If the determiner is "the", modify the following SELECT 

function with DISTINCT, as will be shown by 

examples below. 
Auxiliary Verbs 

An auxiliary verb together with either its immediate noun 
phrase or a non-auxiliary verb determine whether the answer 
should be singular or plural. 

Adjectives 

One grammatical component of the question can be an 
adjective. As shown in FIG. 10, based on the adjective, the 
programming-steps generator either identifies the value of 
an attribute, or identifies an algorithm. The grammatical 
components in the question other than the adjective have 
already selected one or more topic-related tables. 

As shown by the topic-independent semantic table, the 
adjective may identify (Step 350) an attribute. The function 
Attribute-Names({adjective}, table) can retrieve the 
attribute in the table previously selected. The corresponding 
instruction can be: 

for Attribute in Attribute-Names({ adjective}, Table) 

do 

SELECT . . . 
FROM Table 

WHERE Attributc-{adjective} 
// or "Where the attribute in the table is equal to the 
adjective." 

endfor 

As an example, the noun phrase is "a red apple." The noun 
"apple" can be associated with a table known as FRUIT, and 
the Attribute-Names(red, FRUIT) yield the attribute "color." 
The adjective "red" is interpreted: 

WHERE coloured " 

If there is a sequence of such adjectives, all of them can 
apply to the same table. The WHERE clause would be a 
conjunction of the adjectives, such as: 
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WHERE 

for Attribute 1 in Attribute-Names({adjectivel}, Table) 
do 

for Attribute2 in Attribute -Names({adjective2}, 
Table) 

do 

SELECT . . . 

FROM Table 
WHERE Attributelofadjectivel} 
and Attribute2»{adjective2} 
endfor 
endfor 

An adjective can refer to an algorithm, as identified by the 
topic-independent semantic table. Grammatical components 
in the question other than the component that is the adjective 
have selected one or more topic-related tables. As shown in 
the topic-independent semantic table, the adjective identifies 
(Step 3S2) one or more attributes in those tables. Then the 
algorithm operates (Step 3S4) on one or more data in those 
attributes. 

As an example, the adjective is "first." The topic- 
independent semantic table indicates lhat the adjective is an 
algorithm sorting a list of data in ascending order; the table 
also identifies the data in one or more attributes in one or 
more topic-related tables. For each attribute identified, after 
sorting its data, the first value will be the result. For example, 
the question is "Who is the first President" The table 
identified is the President table. The attribute whose data are 
to be sorted is the "date" attribute, which identifies the time 
each President was elected. The instruction corresponding to 
the adjective "first" can be as follows: 

for Attribute in Attribute-Names( first, Table) 

do 

SELECT . . . 
FROM Table 

ORDER BY Attribute ASC 
endfor 

The symbol ASC denotes ascending. 

Similarly, if the adjective is "last," then the attribute 
whose data are ordered is the same, but the data are sorted 
in a descending manner. The corresponding instruction can 
be as follows: 

for Attribute in Attribute-Names(last, Table) 

do 

SELECT . . . 
FROM Table 

ORDER BY Attribute DESC 
en ifor 

The symbol DESC denotes descending. 

Another example on adjective is the word, "immediate." 
Its interpretation depends on the word it modifies. In one 
example, if the word modified is "action," the word "imme- 
diate" has the same effect as the word, "first;" if the word 
modified is "cause," the word "immediate" has the same 
effect as the word "last." 

There can be a sequence of adjectives. Then, the above 
analysis is applied in the same order as the occurrence of the 
adjectives. 

Preposition 

One grammatical component can be a preposition. A 
preposition can modify its previous noun phrase or verb, 
such as by operating on them through an algorithm identified 
in the topic-independent semantic table. Under some 
situations, with one or more tables selected by at least one 
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grammatical component in the question other than the 
component that is the preposition, the algorithm identified 
operates on data or values in the one or more selected tables. 

Under some other situations, for example, due to the 
5 prepositions 'of and 'in*, the programming-steps generator 
processes the grammatical component succeeding the prepo- 
sition before the grammatical component preceding. 

For another example, the preposition 'before' can modify 
the WHERE clause with a comparison on time: 
10 {time of preceding event} <{ time of succeeding event} 
Programming-Steps Executor 

The executor 112 executes at least one set of instructions 
generated from one grammatical component to at least 
access data from the database to generate an answer for the 
15 question, if there is one. 

In one embodiment, after the programming-steps genera- 
tor 108 generates a set of instructions, the programming- 
steps executor 112 executes them. The set may be generated 
from one grammatical component. This process repeats until 
20 all sets are generated and executed to answer the question. 
For at least one set of instructions, the executor 112 accesses 
data from one or more topic-related tables identified by the 
instructions. In another embodiment, all the instructions are 
generated; then the program executor 112 runs the 
25 instructions, which include accessing data from one or more 
topic-related tables identified by the instructions, and pro- 
cessing those data for generating the answer to the natural- 
language question. 

In the appendix, there are a number of examples of 
30 instructions illustrating the present invention. They gener- 
ated to answer different types of grammatically-context-free 
questions. 

Ambiguous Questions 

In the present invention, the grammatical structure ana- 

35 lyzer 102 may decide that the natural-language question 
cannot be parsed into grammatical components based on the 
pre-defined context-free grammatical structure. For 
example, the grammatical components of the question can- 
not fit into the pre-defined structure. Then the question is 

40 considered ambiguous, and an answer cannot be generated. 
Ambiguity may be due to a number of reasons. For 
example, the question may contain words with non-unique 
grammatical meaning the question may contain words not in 
the grammatical table, or the grammatical structure of the 

45 question is different from the pre-defined grammatical struc- 
ture. FIG. 11 shows different approaches to resolve the 
ambiguity. 

The grammatical structure analyzer can decide that a 
word can be of more than one grammatical meaning, such as 

so it can be a noun and a verb. In one embodiment, the analyzer 
produces (Step 402) an answer for each meaning and ignores 
those meaning with no answer. In another embodiment, the 
analyzer asks (Step 400) the user to identify the correct 
grammatical meaning. 

55 For example, the question is: "When was the Persian Gulf 
war?" The word "war" can be a noun or a verb. In one 
embodiment, the analyzer asks the user whether the word 
"war" is a noun or a verb. Based on the user's response, the 
question is analyzed. In another embodiment, the analyzer 

60 generates answers to both the question that treats the word 
"war" as a verb, and the question that treats the word "war" 
as a noun. Both answers, if available, are presented to the 
user. 

If the grammatical structure analyzer decides that the 
65 question contains one or more words not in the grammatical 
table, in one embodiment, the analyzer removes (Step 404) 
the un-recognized word and processes the remaining words 
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in the question. In another embodiment, the analyzer asks 
(Step 406) the user for a different word. The analyzer might 
assume that the word is miss-spelled, and ask the user to 
correct it; the analyzer might replace (Step 408) the 
un-recognized word with a word in the grammatical table 5 
most similar to or with minimum number of different 
characters from the un-recognized word. The analyzer then 
presents (step 410) the matched word to the user to ask if that 
is the right word. A list of matched words may be presented 
for the user to select. 10 

For example, the question is: "What exactly are the Bills 
of Rigbt?"The word "exactly" is an adverb and is not in the 
grammatical table. The word is dropped, and the question, 
satisfying the grammatical structure, is analyzed. In another 
example, the question is: "What is the Bill of Right?" Here, 15 
the "Bill of Right" should be the "Bills of Right." The 
analyzer can ask the user to spell the "Bill of Right" again; 
or the analyzer can find the term closest in spelling to the 
un-recognized term, and identify the term to be the "Bills of 
Right". The identified word is presented to the user to ask if 2 o 
that is the right spelling. 

In the present invention, the grammatical structure of the 
question entered may be different from the one or more 
pre-defined context-free grammatical structures in the sys- 
tem. 25 

In one embodiment, a non-essential grammatical compo- 
nent is missing from the question. A grammatical component 
is non-essential if that grammatical component can be 
removed from the question without changing the answer to 
the question. For example, an auxiliary verb in certain 30 
condition can be non-essential. One approach to solve this 
problem is to ignore (Step 412) the missing grammatical 
component in generating the answer to the question. Another 
approach is to add the missing non-essential grammatical 
component back into the question, and present to the user 35 
asking if that is correct. For example, the question is: "When 
President Nixon resign?" An auxiliary verb is expected after 
the word "When"; such a word is entered into the question, 
which is then submitted to the user for approval. 

In another embodiment, the user is suggested to re-enter 40 
(Step 414) the question with advice as to the appropriate 
question structure. One advice is to ask the user to re-enter 
the question based on the pre-defined structure, such as 
using one of the i-pronouns in the pre-defined grammatical 
structure. This can be done, for example, by citing a list of 45 
acceptable i-pronouns, and a list of model questions using 
the i-pronouns as examples. Another advice is to identify 
nouns and non -auxiliary verbs, if any, in the question, and to 
ask the user which of the identified word or words he wants 
to know more about. Then it would be up to the user to select 50 
the one he wants. In a farther embodiment, the identified 
word or words are fit into alternative grammatical structures, 
and the user is asked to select one structure out of the list of 
suggested structures. 

As an example, the question is: "Do you know when 55 
President Nixon resign?" Assume that such a question does 
not fit the pre-defined grammatical structure. The user is 
suggested to re-enter the question using one of the following 
i-pronouns: What, when, where, why and who. In another 
embodiment, the noun and the auxiliary verb are identified, 60 
and they are "President Nixon resign." The user is asked, 
"You want to know about President Nixon resign?" In yet 
another embodiment, the identified words are fit into the 
following question formats, and it would be up the user to 
select one, for example: 65 

What does President Nixon resign? 

When does President Nixon resign? 
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Where does President Nixon resign? 

Why does President Nixon resign? 

Who does President Nixon resign? 

As another example, the question is: "Is there a reason 
why President Clinton sent troops to Bosnia?" Assume that 
the question does not fit the pre-defined grammatical struc- 
ture. In one embodiment, the user is suggested to re-enter the 
question using one of the i-pronouns in the pre-defined 
grammatical structure. In another embodiment, the nouns 
and the non-auxiliary verbs, "President Clinton", "troops" 
"send" and "Bosnia" are identified. Then the user is asked to 
select one or more of the following questions: 

Do you want to know about President Clinton? 

Do you want to know about troops? 

Do you want to know about Bosnia? 
Also, the answer generator 100 can present suggestions to 
the user on ways to rephrase the original question based on 
the noun and the non-auxiliary verbs. It would then be up to 
the user to select the one he wants. 

In certain situation, the present invention does not have 
any answer. As an example, the grammatical table does not 
have some essential terms X in the question. Then, the 
present invention can return the following message: 

Sorry, I do not know anything about X. You may want to 
check with your instructor. If you wish, I will inform 
your instructor your question. {Click here to inform 
your teacher} 

If the user clicks at the designated area, his last question will 
be automatically sent as an electronic mail to an instructor 
who can answer the user directly. This instructor may be 
previously selected by the user, or the guardian of the user. 
Questions Matching Engine 

The embodiment shown in FIG. 2 can answer an infinite 
number of questions. 

FIG. 12 illustrates another embodiment 500 of the answer 
generator, which provides answers to a finite number of 
questions, but requires fewer steps to generate answers as 
compared to the embodiment shown in FIG. 2. Also, the 
answer generator 500 can answer non-natural-language 
questions, and grammatically-context-t-dependent ques- 
tions. In this embodiment, the database 106 includes a 
questions table, which contains many questions, each with 
its corresponding answer. A question matching engine 529 
compares the question entered with questions in the data- 
base. An answer retriever 539 retrieves the answer to the 
question in the database that matches the entered question. 

FIG. 13 shows one set of steps 550 for the present 
embodiment. The question matching engine 529 compares 
(Step 552) the entered question with questions in the data- 
base 106. If there is a match with any one of them, the 
answer retriever 639 retrieves (Step 54) the answer corre- 
sponding to the matched question. If no question in the table 
matches the input question, the answer generator 500 might 
use one of the approaches discussed in the ambiguous 
questions section to answer the question. 

In another embodiment, the question entered is a natural - 
language question. The matching engine 529 compares the 
grammatical components of the natural language question 
with components of the questions in the database 106. 

A further embodiment includes an essential-components 
extractor, which extracts essential components from the 
natural -language question entered. Only essential compo- 
nents are compared with the pre -stored questions, which 
have essential components. If there is a match, the answer to 
the corresponding matched question is retrieved and is 
presented to the student. As an example, an auxiliary verb is 
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a non-essential components. The extractor strips off the 
auxiliary verb from the question to allow the matching 
engine 529 to compare the rest of the components. 

In yet another embodiment, the question entered is a 
grammatically context-free question. s 

The answer generator 100 shown in FIG. 2 can be 
combined with the answer generator 500 shown in FIG. 12. 
The question entered is first analyzed by the answer gen- 
erator 100 shown in FIG. 2. If the question cannot be parsed 
into its grammatical components based on the pre-defined 10 
structure, then the question is passed to the answer generator 
500 shown in FIG. 12. If that answer generator also cannot 
find a match in the questions table, the question will be 
considered as an ambiguous question to be resolved by 
approaches discussed in the ambiguous questions section. 15 
Question Comparator 

In one embodiment, the comparator 60 compares the 
question just entered with one or more questions previously 
entered by the user to determine his understanding level in 
the subject. This can be done for example by the comparator 20 
60 comparing the grammatical components of the questions. 
In one embodiment, non-essential components are 
de-emphasized. Two questions are considered identical if 
their essential components are identical. Words are consid- 
ered identical to its synonyms, as defined by the topic- 25 
independent-semantic table in the database. If the user has 
asked the same question more than once, his understanding 
level is low in the areas covered by the question. The more 
times he asked the same question, the less he understands the 
area covered by the question. 30 

In another embodiment, the comparator 60 counts the 
total number of occurrence of every interrogative pronoun, 
every noun and every non-auxiliary verb in the question just 
asked based on all the questions he previously asked. If the 
questions are: 35 

Just entered: What is the derivative of sin(x+4) with 
respect to x? 

Previously asked: What is the derivative of cos(2*x)*sin 
(x+4) with respect to x? 
the comparator 60 has the following word counts: 
what: twice, 
derivative: twice, 
sin: twice, 

x: 4 times. 45 
The noun x is known as an indeterminant, which is a 
non-essential word; it is not essential to determine his 
understanding level. In one embodiment, they are ignored in 
word counts. 

If the questions are: 50 
Just asked: When did President Clinton become presi- 
dent? 

Previously asked: How many terms have President 
Clinton served? 

the comparator 60 has the following word counts: 55 
When: once, 
President Clinton: twice, 
president: once, 

become: once. 60 
There is also a question count for the question just asked. 
That count sums the word counts of the words in the 
question, and divides that sum by the number of essential 
words in the question: 

Question count-Sum (Word counts)/(# of essential words 65 
in the question) 
The division normalizes the question count. 



40 



Based on the above metrics, the user's understanding 
level in the area covered by the question is low if the 
question has a high question count. 

In another embodiment, the word count and the question 
count also consider time as a factor. The user might have 
asked a question similar to one he just asked long time ago. 
In order for the word count and the question count to reflect 
his degree of forgetfulness, the system uses an effective 
word count, an effective question count, and time-stamps. 
The effective word count adjusts the word count by a time 
factor. One equation for the effective word count of a word 
is: 

Effective word couol-l+(word couDt)*c/exp(CurTent-time-8tamp- 
LuMimc-stamp), 

where: 

c is a positive constant between 0 and 1; 

exp is the exponential function, 

time-stamps are measured in units of time, 
such as every ten minutes is one unit, 
but with the units removed in the equation; 

current-time-stamp denotes the time 
when the user just asked the question with the word; 
and 

last-time-stamp denotes the time 
when the user asked a question with the word 
immediately before the current-time-stamp. 
The effective question count sums the effective word counts 
of the essential words in the question, and divides that sum 
by the number of essential words in the question. 

Effective question count«Sum (Effective word counts)/(# 
of essential words in the question) 
The division normalizes the effective question count. 

Again, based on the above metrics, the user's understand- 
ing level in the area covered by the question is low if the 
question has a high effective question count. 

In a further embodiment, the comparator 60 also includes 
a word-significance table, which indicates the significance 
of words used in a question. Every word in the subject has 
a significance factor ranging from 0 to 1. For example, the 
non-essential components, just like the indeterminants in 
mathematics, have a significance factor of 0; and the inter- 
rogative pronoun "why" has a higher significance factor 
relative to the interrogative pronoun "what." In one 
embodiment, before the comparator 60 sums the word 
counts iO generate the question counts, each word count is 
multiplied by its corresponding significance factor. 

In another embodiment, based on the magnitude of the 
question count, the comparator 60 may test the user. The test 
results further indicate the user's understanding level in 
areas covered by the question. Generating a test in a certain 
area should be obvious to those skilled in the art and v ill not 
be further described. 

In yet another embodiment, based on the user's under- 
standing level, the comparator 60 sends a message to the 
study-materials generator 52 to retrieve study materials for 
him. In one approach, the less he understands a certain area, 
the more detailed is the study materials to be presented to 
him. In another approach, the less he understands a certain 
area, the lower the level of difficulty is the study materials 
to be presented to him. For example, if the user is very weak 
in fractions, then the presenter 120 presents study materials 
on level 1 of fractions to him. Generating and retrieving 
study materials with different degrees of difficulties and 
different amount of detail should be obvious to those skilled 
in the art, and will not be further described. 

If the user still asks the same question after the system has 
presented to him detailed study materials, the answer gen- 
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erator 100 may ask him to consult an instructor. In one 2. A computer-implemented method as recited in claim 1, 

embodiment, the database 106 contains a list of instructors wherein said method further comprises: 

for different areas of the subject. With permission from the identifying additional informational materials related to 

user, the answer generator 100 may contact one or more me ntluraWangluge ques tion; and 

instructors through electronic mail or other means, with the 5 „. .... . ?, . . . , . . 

question sent to the instructor. The instructor can contact the acndu « informational materials from the 

user directly. server system to the client system via the Internet. 

After reading the answer to his question, the user might 3. A computer-implemented method as recited in claim 2, 
ask another question, and the process of answering question wherein said sending of the additional informational mate- 
repeats, rials k performed when requested by the user. 
Filling Gaps of Misunderstanding 10 4. A computer-implemented method as recited in claim 2, 

This invention is also applicable to filling gaps of mis- wherein said sending of the additional informational mate- 

understanding when the user is working on a subject. FIG. rials is automatically performed substantially simultaneous 

14 shows one such embodiment 600. First, the system 50 with said sending of the response, 

generates (Step 602) study materials on a subject, and 5. A computer-implemented method as recited in claim 1, 

presents (Step 604) the study materials to him. The subject 15 wherein the grammatical and semantic processing uses at 

can be a novel he wants to read on the system. The novel least one grammatical rule and at least one semantic rule, 

may have many individuals. He gets confused on their 6. A computer-implemented method as recited in claim 1, 

names, and begins to lose interest in the novel. Clarifying the wherein said method further comprises: 

identities might revive his interest in the novel. Similarly, a referral fo me m ^ 

K T ^.^Y * t0 W* h u mCanm l V 20 to "turd language question be referred to a human 

term, such as capital asset pacing model. He can ask the a 

system a question. The system 50 retrieves (Step 606) the representative; and 

question entered, and generates (Step 608) an answer to the enabling a human representative to respond to the natural 

question. The presenter 120 presents (Step 610) the answer language question. 

to him. Filling gaps of misunderstanding is very important 2 s ^' A computer-implemented method as recited in claim 6, 

in learning. Typically, the user gets confused and loses wherein the natural language question is sent to the human 

interest in the study materials as misunderstanding representative without the need for the user to re-enter the 

increases. After the system has answered his question, the question into the client system. 

process repeats with the system 50 generating study mate- 8. A computer-implemented method as recited in claim 6, 

rials for the user. In one embodiment, the process continues 3Q wherein the human representative is specifically for the 

until the user has mastered the entire subject. subject, and a different human representative may be 

In one embodiment, after working on the subject for some enabled for a different subject, 
time, the user stops. The database stores the time he stops, 9. A computer-implemented method as recited in claim 1, 
with his identity, and the location where he terminates wherein said method further comprises: 
learning the subject. Next time, when the users enters the 35 referring the natural language question to a human rep- 
answer generator 100 to learn the same subject again, the resentative to respond to the natural language question, 
answer generator 100 re-starts the process from where he 10. A computer-implemented method as recited in claim 
ended last time. In another embodiment, the answer genera- 9, wherein the natural language question is sent to the human 
tor 100 asks him if he wants to re-start from where he ended representative without the need for the user to re-enter the 
or to re-start from another part of the subject. It would be up question into the client system. 

to him to decide. 11, A computer-implemented method as recited in claim 

Other embodiments of the invention will be apparent to 9, wherein the human representative is specifically for the 

those skilled in the art from a consideration of this specifi- subject, and a different human representative may be 

cation or practice of the invention disclosed herein. It is enabled for a different subject. 

intended that the specification and examples be considered ^ 12. A computer-implemented method as recited in claim 

as exemplary only, with the true scope and spirit of the 1, wherein said method further comprises: 

invention being indicated by the following claims, determining whether clarification of the natural language 

What is claimed is: question is desirable; and 

1 A computer-implemented method for providing a user prompting the user to clarify the natural language ques- 

with information, said method comprising: 5o ^ when clarificatioQ of thc natural languagc que M stioD 

sending informational materials from a server system to a ^ determined to be desirable, 

client system via the Internet so as to inform a user of 13 A computer-implemented method as recited in claim 

the client system about a subject; 12 , wherein the user is asked one or more questions when 

subsequently receiving, at the server system, a request to prompting the user to clarify. 

respond to a natural-language question asked by the 55 14. a computer-implemented method as recited in claim 

user, the natural language question being asked by the 1, wherein said method further comprises: 

user at the client system, and the natural language sending at least two questions to the client system; and 

question being associated with the subject; receiving, at the server system, at least an indication of a 

determining, at the server system, a response to the selection by the user of one of the questions sent by the 

natural-language question, said determining operating 60 server system t0 be responded, 

to at least analyze the natural-language question using 15. A computer-implemented method as recited in claim 

at least grammatical and semantic processing; and i, wherein the response is an answer to the natural-language 

sending the response to the natural-language question question. 

from the server system to the client system via the 16. A computer-implemented method as recited in claim 

Internet; 65 ^ wherein said determining operates to transform at least a 

wherein the subject relates to a product that the user is portion of the natural-language question into at least one 

interested in. instruction. 
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17. A computer- implemented method as recited in claim 
16, wherein the server system has an information database 
coupled thereto, and wherein the instruction is a query for 
the database. 

18. A computer-implemented method as recited in claim 
1, wherein said determining is independent of at least one 
word in the natural-language question, and the response 
depends on a correct spelling of at least one misspelled word 
in the natural -language question. 

19. A computer-implemented method as recited in claim 
1, wherein the subject relates to a product the user is 
interested in obtaining. 

20. A computer-implemented method as recited in claim 
1, wherein the subject relates to a product the user is 
interested in using. 

21. A computer- implemented method as recited in claim 
1, wherein said method further comprises having at least a 
second response available for the user, and allowing the user 
to select one or more of the responses for the natural- 
language question. 

22. A computer-implemented method as recited in claim 
1, wherein said determining produces a plurality of 
responses, and 

wherein said sending operates to send the plurality of 
responses from the server system to the client system. 

23. A computer- implemented method as recited in claim 
22, wherein at least one of the responses is a question. 

24. A computer- implemented method as recited in claim 
22, wherein said method further comprises: 

selecting one of the plurality of responses for which a 

further response is desired. 
23. A computer-implemented method as recited in claim 
24, wherein at least one of the responses is a question. 

26. A computer-implemented method as recited in claim 
1, wherein said method further comprises considering at 
least one additional input entered by the user in determining 
the response. 

27. A computer-implemented method as recited in claim 

26, wherein the additional input is a response to an inquiry 
from the server system. 

28. A computer-implemented method as recited in claim 

27, wherein the additional input is for clarifying the natural 
language question. 

29. A computer-implemented method as recited in claim 
26, wherein the additional input is also a question from the 
user. 

30. A computer-implemented method as recited in claim 
1, wherein the server system keeps track of a unique 
identifier of the user. 

31. A computer-implemented method as recited in claim 
1, wherein said method further comprises: 

adapting in real-time to respond to questions by consid- 
ering at least one additional input. 

32. A computer-implemented method for providing a user 
with information, said method comprising: 

sending informational materials from a server system to a 
client system via the Internet so as to inform a user of 
the client system about a subject; 

subsequently receiving, at the server system, a request to 
respond to a natural-language question asked by the 
user, the natural language question being asked by the 
user at the client system, and the natural language 
question being associated with the subject; 

determining, at the server system, a response to the 
natural-language question, said determining operating 
to at least analyze the natural-language question using 
at least grammatical and semantic processing; 
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sending the response to the natural-language question 
from the server system to the client system via the 
Internet; 

receiving a referral request from the user requesting that 
5 the natural language question be referred to a human 
representative; and 
enabling a human representative to respond to the natural 

language question, 
wherein the subject relates to a product that the user is 
10 interested in using. 

33. A computer-implemented method as recited in claim 
32, wherein said method further comprises having at least a 
second response available for the user, and allowing the user 
to select one or more of the responses for the natural- 

15 language question. 

34. A computer-implemented method as recited in claim 
32, wherein said determining is independent of at least one 
word in the natural-language question, and the response 
depends on a correct spelling of at least one misspelled word 
in the natural-language question. 

20 33. A computer-implemented method as recited in claim 
32, wherein said determining operates to transform at least 
a portion of the natural-language question into at least one 
instruction. 

36. A computer-implemented method as recited in claim 
25 33, wherein the server system has an information database 

coupled thereto, and wherein the instruction is a query for 
the database. 

37. A computer-implemented method as recited in claim 
32, wherein said method further comprises: 

30 sending at least two questions to the client system, and 
receiving, at the server system, at least an indication of a 
selection by the user of one of the questions sent by the 
server system to be utilized as the natural language 

35 question being asked by the user. 

38. A computer-implemented method as recited in claim 
32, wherein said method further comprises considering at 
least one additional input entered by the user in determining 
the response. 

^ 39. A computer-implemented method as recited in claim 
38, wherein the additional input is a response to an inquiry 
from the server system. 

40. A computer-implemented method as recited in claim 
38, wherein the additional input is also a question from the 
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41. A computer-implemented method as recited in claim 
32, wherein the server system keeps track of a unique 
identifier of the user. 

42. A computer- implemented method for providing a user 
5o with information, said method comprising: 

sending informational materials from a server system to a 
client system via the Internet so as to inform a user of 
the client system about a subject; 

subsequently receiving, at the server system, a request to 
55 respond to a natural-language question asked by the 
user, the natural language question being asked by the 
user at the client system, and the natural language 
question being associated with the subject; 

determining, at the server system, a response to the 
60 natural-language question, said determining operating 
to at least analyze the natural-language question using 
at least grammatical and semantic processing; 

sending the response to the natural-language question 
from the server system to the client system via the 
65 Internet; 

identifying additional informational materials related to 
the natural-language question; and 



02/26/2004, EAST Version: 1.4.1 



